apache / flink-cdc

Flink CDC is a streaming data integration tool
https://nightlies.apache.org/flink/flink-cdc-docs-stable
Apache License 2.0
5.74k stars 1.95k forks source link

How to do MySQL to MySQL pipeline #3140

Closed ysmintor closed 7 months ago

ysmintor commented 8 months ago

Hi, I just found out demo of MySQL to Doris, But I want to do a whole sync database of MySQL to another MySQL. flink-cdc-pipeline-connector-mysql is just for source. How about the sink still is MySQL?

Cloud anyone tell me some information to implement.

ihadoop commented 8 months ago

you can create sink mysql table

CREATE TABLE mysql_sink (
  `id`  bigint ,
  `s_name` varchar(100)  ,
  `s_int` bigint,
 PRIMARY KEY(id) NOT ENFORCED 
) WITH (
 'connector' = 'jdbc',
'driver' = 'com.mysql.cj.jdbc.Driver',
'url' = 'jdbc:mysql://192.168.1.15:3306/cdc?serverTimezone=UTC&useSSL=false',
 'username' = 'root',
 'password' = '111111',
 'table-name' = 't_t'
);

insert into mysql_sink select * from cdc_table
ysmintor commented 8 months ago

@ihadoop Thanks for you reply. But your method means that I have to create every table which is to not a suit case like pipeline way to sync MySQL database into Doris with each table has the corresponding tables in Doris database.

KiwiGeorge commented 8 months ago

Owner said mysql2mysql still not support even though there is source code

ysmintor commented 8 months ago

Owner said mysql2mysql still not support even though there is source code

Oh, Thank you. I have found some DataStream method to handle which take CDC DDL to create corelated table in sink. I also found third party Dinky which is a streaming process platform based on Flink that already implement MySQL to MySQL in their way. So I hope at someday FlinkCDC 3.0+ will implement those features.

PatrickRen commented 7 months ago

Closing this issue as it has been migrated to Apache Jira.