search

databricks / iceberg-kafka-connect

Apache License 2.0
219 stars 49 forks source link

Add a new configuration parameter for mapping Kafka topics to tables #206

Open igorvoltaic opened 8 months ago

igorvoltaic commented 8 months ago

Add a new configuration parameter which maps Kafka topics and Iceberg tables:

E.g. topics.to.tables.mapping=some_topic0:table_name0,some_topic1:table_name1

Similar approach implemented in SnowflakeSink, ClickhouseSink, Aiven JdbcSink

almir-magazord commented 8 months ago

That would be very useful! :crossed_fingers:

stefan-ce commented 5 months ago

Would be extremely useful especially when using the Debezium source plugin and outbox router. You just need one config file for the Debezium source connector plugin, but one config file per topic (aggregate root) and always add one more Iceberg sink connector config when you are adding one more aggregate root in your application to the outbox.

If this was implemented in a way that the table name would be taken from the topic name (or part of it) it would even work automatically without any config changes in Kafka Connect at all.