search

questdb / kafka-questdb-connector

QuestDB connector for Kafka.
https://github.com/questdb/kafka-questdb-connector/releases/
Apache License 2.0
31 stars 5 forks source link

Configuration file #3

Open CSharpDummy opened 2 years ago

CSharpDummy commented 2 years ago

I'm a bit lost about the configuration file. Can I just define in the docker build process and copy it into the right directory? Also not sure which directory to store it.

I am trying to use docker compose and set up questdb, the questdb kafka conenctor and redpanda as kafka broker.

iamkucuk commented 2 years ago

You may want to watch the video in here: https://www.youtube.com/watch?v=Jkcp28ki82k

jerrinot commented 2 years ago

Hello @CSharpDummy,

indeed it's a bit confusing. In short: Kafka Connect has 2 modes:

  1. Standalone
  2. Distributed

The Standalone mode indeed allows you to define a configuration property file and pass the file as an argument to Kafka Connect. This is described at https://questdb.io/docs/third-party-tools/kafka/questdb-kafka/

However, the Standalone mode has drawbacks in terms of scalability and fault tolerance - there can be only one JVM running the connector - when this crash then you have to restart it.

The Distributed mode has better fault-tolerance properties and also it allows to run the same connect in multiple JVMs. Unfortunately, it's a bit awkward in terms of configuration - you have to start connectors via the REST API.

The video shared by @iamkucuk looks good, I assume it'll explain these concepts into greater details.

I could also add a Dockerized code sample with the standalone mode - given the high performance of QuestDB connector there is not much need for horizontal scalability and failures can be handled on different levels, eg. Kubernetes. So thanks for creating this issue!

CSharpDummy commented 2 years ago

Ah yes that makes sense. So the connector spans over the cluster.

jerrinot commented 2 years ago

@CSharpDummy exactly. there is a bit mismatch between the Distributed mode and the way how container orchestrator work. but an individual connector cannot change that.

I'm keeping this issue open as a reminder to add a Standalone code example.