search

ClickHouse / clickhouse-kafka-connect

ClickHouse Kafka Connector
Apache License 2.0
147 stars 40 forks source link

Support automatic table schema evolution #277

Open mlivirov opened 9 months ago

mlivirov commented 9 months ago

Description

In my organization we use clickhouse as analitical database that gets it's data from multiple sources via Kafka. Domain models change from time to time as business changes and we found a need to reflect those changes in the clickhouse manually a little inconvenient.

Would be awesome if this connector could propagate changes in schemas to the clickhouse tables automatically.

Additional context Compatibility modes in kafka:
https://docs.confluent.io/cloud/current/sr/fundamentals/schema-evolution.html#compatibility-types

JDBC connector that has this functionality: https://github.com/confluentinc/kafka-connect-jdbc

Paultagoras commented 3 months ago

Hi @mlivirov ! So this seems like it would have a race condition, if there are multiple tasks running - is that not the case with the jdbc connector mentioned?

mlivirov commented 3 months ago

Not sure how it's done in jdbc connector. Wouldn't it be possible to use separate KeeperMap table to synchronize tasks?