[SUPPORT] Deep integration flink cdc？

apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.

https://hudi.apache.org/

Apache License 2.0

5.35k stars 2.42k forks source link

[SUPPORT] Deep integration flink cdc？ #9873

Open melin opened 11 months ago

melin commented 11 months ago

Supports data synchronization between multiple tables or databases, adding application scenarios. @danny0405

danny0405 commented 11 months ago

We have no plan for that now, do you have intreast to contribute?

melin commented 10 months ago

Both paimon and hudi support streaming data writing well. There is a strong demand for real-time synchronization of data from relational databases to data lakes. paimon CDC Ingestion supports database or multi-table synchronization, which will greatly facilitate users to synchronize data to the data lake. The cost of solving such problems with hudi is still very high.

danny0405 commented 10 months ago

Yeah, but the complexity of Hudi does not really allows too many tables to sync in one pipeline, create a utilitiy makes sense to me if you think it is feasible, do you have intreast to controbute one, it should be easy to do that.