itinycheng / flink-connector-clickhouse

Flink SQL connector for ClickHouse. Support ClickHouseCatalog and read/write primary data, maps, arrays to clickhouse.
Apache License 2.0
346 stars 148 forks source link

sink.update-strategy不生效 #99

Closed xixer closed 8 months ago

xixer commented 9 months ago

flink 1.16.2,版本库master版本的connector

在flink sql中(inert into .. select .. a join b..)未指定或指定sink.update-strategy为update,通过cdc最终写入clickhouse均为insert(重复行)

(ddl中指定了主键

itinycheng commented 9 months ago

@xixer

这个配置我之前测试过是有效的; 可以本地起一个测试的Demo,断点看下原因; 具体实现在ClickHouseUpsertExecutor.addValueToBatch内;

itinycheng commented 9 months ago

不过不建议使用这个功能,因为底层是通过alter..update..进行数据更新的,这个操作比较重; 建议在ClickHouse端 使用 MergeTree + optimize table 来进行数据distinct;