search

itinycheng / flink-connector-clickhouse

Flink SQL connector for ClickHouse. Support ClickHouseCatalog and read/write primary data, maps, arrays to clickhouse.
Apache License 2.0
374 stars 156 forks source link

How to guarantee exactly-once semantics? #20

Closed LuPan2015 closed 2 years ago

LuPan2015 commented 2 years ago

Since clickhouse does not support transactions, how to ensure exactly-once semantics when flink writes？

itinycheng commented 2 years ago

Since clickhouse does not support transactions, how to ensure exactly-once semantics when flink writes？

The project can guarantee idempotent when using the MergeTree engine but not exactly-once. In my view, flink can keep end-to-end exactly-once when both endpoints support transactions.