search

databendcloud / flink-connector-databend

Flink SQL connector for Databend.
Apache License 2.0
4 stars 1 forks source link

[Feature]: Support MySQL full-database synchronization or multi-table synchronization. #22

Open hantmac opened 11 months ago

hantmac commented 11 months ago
hantmac commented 9 months ago

This can based on the Flink DataStream package, which can implement a Flink task for submitting a whole database/multi-table synchronization by executing the command line. The main process is as follows:

  1. Obtain the MySQL link information, and read the table that needs to be synchronized this time according to the configuration
  2. Determine whether the table exists in Databend, if not, create the table
  3. Obtain CDC upstream data, and use the side output stream for distribution according to the table name
  4. Create a DatabendSink for each stream.
  5. Submit FlinkJob