Investigate consumer group rebalance

housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse

https://housepower.github.io/clickhouse_sinker

Apache License 2.0

515 stars 118 forks source link

Closed yuzhichang closed 3 years ago

yuzhichang commented 3 years ago

Kafka consumer group rebalance causes multiple clickhouse_sinker instances writing duplicated messages to ClickHouse. Some possible fix:

Catch and handle such events
clickhouse_sinker instances coordinate with each other directly. Don't use Kafka consumer group.

yuzhichang commented 3 years ago

7984fb9024dd5694343d4b8220d8e4f8a810b0df leverage sarama generation cleanup callback to ensure exatctly-once when rebalance occur.