search

hpgrahsl / kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector
Apache License 2.0
153 stars 60 forks source link

Support for topics wildcards #80

Closed nazr closed 5 years ago

nazr commented 5 years ago

Hi, since multiple topics are supported, it would be useful to support wildcards in "topics" as well.

nazr commented 5 years ago

Perhaps it's already supported? https://issues.apache.org/jira/browse/KAFKA-3073

hpgrahsl commented 5 years ago

@nazr you should be able to specify a regexp for the topics property in the sink configuration. this then causes connect to subscribe to all of them. what's important to keep in mind is that if you happen to need e.g. individually different settings on topic level you should configure the topic specific overrides accordingly. another option would be e.g. to use the empty string for mongodb.collection= which would then fallback to the original kafka topic name to be used as the mongodb collection name. let me know if it works for you as expected.

nazr commented 5 years ago

thanks for the reply! Our requirement is to sink all the topic that match a wildcard definition to a single collection. It sounds like the collection name will be different though, so it won't work?

The list of topics is not known (hence wildcards), so can't configure it on per topic basis.

hpgrahsl commented 5 years ago

you can configure the connector so that it reads multiple kafka topics and sinks them into the same mongodb collection. in fact this behaviour was there from the very beginning where it wasn't possible to specify topic/collection specific configuration options. just try to put your regexp for the topics property and then set the mongodb.collection property to a name of your choice for the sink collection which should receive all records regardless of which topic they originate from. beware that this is typically not recommended in case the structure of the documents is vastly different :)

nazr commented 5 years ago

Thanks! it does work with latest version of everything. It's worth noting that it'll only pick up existing topics at the time the connector is created. Any new topics, even if they match the wildcard definition wouldn't be processed.

hpgrahsl commented 5 years ago

Glad to hear it works for you as expected. Your observation - namely that the connector only picks up topics that exist at creation time - is the current behaviour of kafka connect itself. The topic subscription happens at framework level. Thus it has nothing to do with my sink connector implementation. I you think kafka connect should periodically check for new matching topics to subscribe to after creation of the connector you may file a feature request for kafka connect anytime.

I suggest to close this issue now. THX again for reaching out!