DataLinkDC / dinky

Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.
http://www.dinky.org.cn
Apache License 2.0
3.14k stars 1.16k forks source link

[Feature][Whole database synchronization] Dinky whole database synchronizes mysql to doris, how to automatically create doris tables based on the configured table name suffix and suffix to achieve data synchronization #3807

Closed NCUZK closed 1 month ago

NCUZK commented 1 month ago

Search before asking

Description

当前现状:

  1. 测试发现需要doris侧先手动建表,然后才能同步,如果不创建会出现类似错误 https://github.com/DataLinkDC/dinky/issues/3667

Use case

实现类似flink-doris-connector整库同步效果 https://doris.apache.org/zh-CN/docs/2.0/ecosystem/flink-doris-connector/#mysql-%E5%A4%9A%E8%A1%A8%E5%90%8C%E6%AD%A5%E7%A4%BA%E4%BE%8B

bin/flink run \ -Dexecution.checkpointing.interval=10s \ -Dparallelism.default=1 \ -c org.apache.doris.flink.tools.cdc.CdcTools \ lib/flink-doris-connector-1.16-1.6.1.jar \ mysql-sync-database \ --database test_db \ --mysql-conf hostname=127.0.0.1 \ --mysql-conf port=3306 \ --mysql-conf username=root \ --mysql-conf password=123456 \ --mysql-conf database-name=mysql_db \ --including-tables "tbl1|test.*" \ --sink-conf fenodes=127.0.0.1:8030 \ --sink-conf username=root \ --sink-conf password=123456 \ --sink-conf jdbc-url=jdbc:mysql://127.0.0.1:9030 \ --sink-conf sink.label-prefix=label \ --table-conf replication_num=1 ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Zzm0809 commented 1 month ago

直接使用 execute jar 语法即可 https://www.dinky.org.cn/docs/next/extend/expand_statements/execute_jar

aiwenmo commented 1 month ago

Auto create table only supports mysql. Other database need to extend.