search

hpgrahsl / kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector
Apache License 2.0
153 stars 60 forks source link

[feature request] support multiple topics #52

Closed hpgrahsl closed 6 years ago

hpgrahsl commented 6 years ago

One of the most requested features based on user feedback so far is to have support for storing kafka records from different topics into separate mongodb collections. see e.g. [#45,#48]

there are basically two ways to achieve that:

  1. the naive way is to simply write every record to a collection in mongodb with the same name that's specified as record name in the sink record. problem is that it would most likely not work since we have to expect different kafka topics to contain completely different data and therefore a lot of configuration settings might be different as well.

  2. a bit more complex: enable a more flexible / sophisticated way to define all relevant configuration options on a topic level so that users can fully benefit of this feature. ideally there is an overriding mechanism in place which allows to change settings specifically on a topic-basis. at the same time this means that there should be fallback settings when nothing is defined at the topic-level.

implementing the 2nd option should be the best way to go :)

hpgrahsl commented 6 years ago

FYI @durduliu2009 @SkiaBr23 since you asked for this a while ago -> eventually a first version of this feature is already in development :)

ricristian commented 6 years ago

@hpgrahsl those are great news !!! I look forward for this to have some tests with this new feature !!!

hpgrahsl commented 6 years ago

@durduliu2009 glad to hear you are looking forward to this :)

the current design will be such that you have to at least specify the collection names explicitly, meaning if you don't need to rename them you set the config options to the same names that are used for the "topics" settings.

will roughly look as follows

...
"topics" : "my-topic1,my-topic2,..."
...

will be addressed for the corresponding mongodb collections in the new configuration by

...
"mongodb.collection.my-topic1":"my-topic1",
"mongodb.collection.my-topic2":"my-topic2",
"mongodb.collection. ... ":"...",
...

at the same time all relevant properties can be then configured on a topic/collection level as well.

hpgrahsl commented 6 years ago

FYI @durduliu2009 @SkiaBr23 it's available as of now. Documentation see https://github.com/hpgrahsl/kafka-connect-mongodb#collection-aware-configuration-settings

SkiaBr23 commented 5 years ago

@hpgrahsl Just saw this. Great work on the feature, actually way more helpful than i expected. Kudos!