search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.44k stars 959 forks source link

[Feature] CDC Ingestion supports double-write kafka #2063

Closed MonsterChenzhuo closed 12 months ago

MonsterChenzhuo commented 1 year ago

Search before asking

Motivation

At present, if users have a reuse requirement for CDC (Change Data Capture) synchronized data, we recommend that users first import the data into Kafka (where Kafka data is used for reuse) and then from Kafka to Paimon. Based on this scenario, I am considering taking an action to write to Paimon, while also writing a copy of the changelog to Kafka, which can simplify the synchronization link.

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

JingsongLi commented 12 months ago

It is not recommended. Kafka external log is deprecated.