search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.29k stars 916 forks source link

[Feature] Support Synchronizing from multiple database in CDC sync-database action #1161

Closed yuzelin closed 1 year ago

yuzelin commented 1 year ago

Search before asking

Motivation

Currently, mysql-sync-database only support sync single database. But a common scenario is a table has been split into multiple database, for example, the db1 and db2 both have a table called tbl.

We want support that the two tables can be synchronized into one table, that means we should support synchronizing from multiple database.

Solution

The mysql-sync-table action can set database name with a regular expression to support synchronizing from multiple database. The solution can be applied to sync database action.

Anything else?

No response

Are you willing to submit a PR?

s7monk commented 1 year ago

please assign this to me. @yuzelin

yuzelin commented 1 year ago

Sure, assigned. @s7monk

zhangjun0x01 commented 1 year ago

Do we need to support synchronizing different MySQL tables to the corresponding paimon table, rather than just synchronizing to the same paimon table ?