search

DataLinkDC / dinky

Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.
http://www.dinky.org.cn
Apache License 2.0
3.08k stars 1.13k forks source link

add feature cdcsource sink table name mapping, regular matching modification #3592

Closed soulmz closed 3 months ago

soulmz commented 3 months ago

Purpose of the pull request

cdcssource Sink Feature addition

Brief change log

Verify this pull request

This change added tests and can be verified as follows:

测试的用例:

set 'execution.savepoint.ignore-unclaimed-state' = 'true';
-- 声明一些调优参数 (checkpoint 等相关配置)
set 'execution.checkpointing.checkpoints-after-tasks-finish.enabled' ='true';
-- checkpoint 的间隔设置
set 'execution.checkpointing.interval' = '60 s';
set 'state.savepoints.dir'='file:///flink-data/jxc/savepoints'; -- 目录自行修改
set 'state.checkpoints.dir'= 'file:///flink-data/jxc/checkpoints'; -- 目录自行修改
set 'state.checkpoint-storage'='filesystem';
set 'state.backend.type'='rocksdb';
-- checkpoint 的保留数量
set 'state.checkpoints.num-retained'='10';
-- 使 solt 均匀分布在 各个 TM 上
set 'cluster.evenly-spread-out-slots'='true';

-- jxc 的 ods 同步方案
EXECUTE CDCSOURCE jxc_ods_cdc WITH (
  'connector' = 'mysql-cdc',
  ${database-conf-test},
  'checkpoint' = '60000',
  'scan.startup.mode' = 'initial',
  'parallelism' = '1',
  'table-name' = 'test_jxc\.t_warehouse_sn,test_jxc\.t_warehouse_expiration',
  'sink.connector' = 'jdbc',
  'sink.sink.db'='ods',
  'sink.url' = 'jdbc:mysql://127.0.0.1:3306/ods?characterEncoding=utf-8&useSSL=false',
  'sink.username' = 'root',
  'sink.password' = '123456',
-- 映射表名 原表t_warehouse_sn 替换成 test_warehouse_sn_test ，多张表通过逗号分割
  'sink.table.mapping-routes' = 't_warehouse_sn:test_warehouse_sn_test,t_warehouse_expiration:test_warehouse_expiration_aa',
 -- 正则匹配表名，进行替换
 'sink.table.replace.pattern' = 't_(.*?)_',
 'sink.table.replace.with' = 'ods_$1_',
  'sink.table.lower' = 'true',
  'sink.table-name' = '#{tableName}',
  'sink.driver' = 'com.mysql.jdbc.Driver',
  'sink.sink.buffer-flush.interval' = '3s',
  'sink.sink.buffer-flush.max-rows' = '500',
  'sink.sink.max-retries' = '5',
  'sink.auto.create' = 'true'
);
Zzm0809 commented 3 months ago

可以同时将以上案例新增一个案例文档,在案例文档中描述具体使用方法,及其注意事项

soulmz commented 3 months ago

可以同时将以上案例新增一个案例文档,在案例文档中描述具体使用方法,及其注意事项

好，晚点弄一个案例

soulmz commented 3 months ago

可以同时将以上案例新增一个案例文档,在案例文档中描述具体使用方法,及其注意事项

@Zzm0809 文档已提交。

Zzm0809 commented 3 months ago

单元测试无法通过ci，需要解决一下 。

另外代码中注释等中文需要改为英文，辛苦大佬修改一下。ヾ(≧∇≦谢谢≧∇≦)ノ

soulmz commented 3 months ago

单元测试无法通过ci，需要解决一下 。

另外代码中注释等中文需要改为英文，辛苦大佬修改一下。ヾ(≧∇≦谢谢≧∇≦)ノ

测试用例重新跑了，没问题。 注释也改了。

Zzm0809 commented 3 months ago

需要执行下代码格式化

soulmz commented 3 months ago

需要执行下代码格式化

我本地是可以的。。

> ./mvnw -T 4C -B --no-snapshot-updates clean spotless:check -P flink-all
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Dinky : Real-time Computing Platform                               [pom]
[INFO] Dinky : Flink                                                      [pom]
[INFO] Dinky : Flink 1.14                                                 [jar]
[INFO] Dinky : Flink 1.15                                                 [jar]
[INFO] Dinky : Flink 1.16                                                 [jar]
[INFO] Dinky : Flink 1.17                                                 [jar]
[INFO] Dinky : Flink 1.18                                                 [jar]
[INFO] Dinky : Flink 1.19                                                 [jar]
[INFO] Dinky : Common                                                     [jar]
[INFO] Dinky : Client                                                     [pom]
[INFO] Dinky : Client : Hadoop                                            [jar]
[INFO] Dinky : Client : Base                                              [jar]
[INFO] Dinky : Client 1.14                                                [jar]
[INFO] Dinky : Client 1.15                                                [jar]
[INFO] Dinky : Client 1.16                                                [jar]
[INFO] Dinky : Client 1.17                                                [jar]
[INFO] Dinky : Client 1.18                                                [jar]
[INFO] Dinky : Client 1.19                                                [jar]
[INFO] Dinky : CDC                                                        [pom]
[INFO] Dinky : CDC: Core                                                  [jar]
[INFO] Dinky : Catalog                                                    [pom]
[INFO] Dinky : Catalog : Mysql                                            [pom]
[INFO] Dinky : Catalog : Mysql 1.14                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.15                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.16                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.17                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.18                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.19                                       [jar]
[INFO] Dinky : Gateway                                                    [jar]
[INFO] Dinky : Function                                                   [jar]
[INFO] Dinky : Metadata                                                   [pom]
......
[INFO] --- spotless-maven-plugin:2.27.1:apply (default-cli) @ dinky-assembly ---
[INFO] Spotless apply skipped
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Dinky : Real-time Computing Platform 1.1.0-SNAPSHOT:
[INFO] 
[INFO] Dinky : Real-time Computing Platform ............... SUCCESS [  0.077 s]
[INFO] Dinky : Flink ...................................... SUCCESS [  0.002 s]
[INFO] Dinky : Flink 1.14 ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Flink 1.15 ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Flink 1.16 ................................. SUCCESS [  0.002 s]
[INFO] Dinky : Flink 1.17 ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Flink 1.18 ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Flink 1.19 ................................. SUCCESS [  0.002 s]
[INFO] Dinky : Common ..................................... SUCCESS [  0.002 s]
[INFO] Dinky : Client ..................................... SUCCESS [  0.003 s]
[INFO] Dinky : Client : Hadoop ............................ SUCCESS [  0.001 s]
[INFO] Dinky : Client : Base .............................. SUCCESS [  0.000 s]
[INFO] Dinky : Client 1.14 ................................ SUCCESS [  0.001 s]
[INFO] Dinky : Client 1.15 ................................ SUCCESS [  0.002 s]
[INFO] Dinky : Client 1.16 ................................ SUCCESS [  0.001 s]
[INFO] Dinky : Client 1.17 ................................ SUCCESS [  0.002 s]
[INFO] Dinky : Client 1.18 ................................ SUCCESS [  0.002 s]
[INFO] Dinky : Client 1.19 ................................ SUCCESS [  0.001 s]
[INFO] Dinky : CDC ........................................ SUCCESS [  0.001 s]
[INFO] Dinky : CDC: Core .................................. SUCCESS [  0.000 s]
[INFO] Dinky : CDC: Plus .................................. SUCCESS [  0.001 s]
[INFO] Dinky : Catalog .................................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql ............................ SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.14 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.15 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.16 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.17 ....................... SUCCESS [  0.000 s]
[INFO] Dinky : Catalog : Mysql 1.18 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.19 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Gateway .................................... SUCCESS [  0.002 s]
[INFO] Dinky : Function ................................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata ................................... SUCCESS [  0.000 s]
[INFO] Dinky : Metadata : Base ............................ SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Mysql ........................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Oracle .......................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Clickhouse ...................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Postgresql ...................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Sqlserver ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Doris ........................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Phoenix ......................... SUCCESS [  0.000 s]
[INFO] Dinky : Metadata : Hive ............................ SUCCESS [  0.000 s]
[INFO] Dinky : Metadata : Starrocks ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Presto .......................... SUCCESS [  0.001 s]
[INFO] Dinky : Connectors ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Connector : JDBC 1.14 ...................... SUCCESS [  0.001 s]
[INFO] Dinky : Extends .................................... SUCCESS [  0.000 s]
[INFO] Dinky : Alert ...................................... SUCCESS [  0.001 s]
[INFO] Dinky : Alter : Base ............................... SUCCESS [  0.001 s]
[INFO] Dinky : Alter : DingTalk ........................... SUCCESS [  0.000 s]
[INFO] Dinky : Alter : Wechat ............................. SUCCESS [  0.001 s]
[INFO] Dinky : Alter : FeiShu ............................. SUCCESS [  0.001 s]
[INFO] Dinky : Alter : Email .............................. SUCCESS [  0.001 s]
[INFO] Dinky : Alter : SMS ................................ SUCCESS [  0.000 s]
[INFO] Dinky : Alter : Http ............................... SUCCESS [  0.001 s]
[INFO] Dinky : Daemon ..................................... SUCCESS [  0.001 s]
[INFO] Dinky : Scheduler .................................. SUCCESS [  0.001 s]
[INFO] Dinky : Core ....................................... SUCCESS [  0.001 s]
[INFO] Dinky : App ........................................ SUCCESS [  0.002 s]
[INFO] Dinky : App : Base ................................. SUCCESS [  0.001 s]
[INFO] Dinky : App 1.14 ................................... SUCCESS [  0.001 s]
[INFO] Dinky : App 1.15 ................................... SUCCESS [  0.001 s]
[INFO] Dinky : App 1.16 ................................... SUCCESS [  0.001 s]
[INFO] Dinky : App 1.17 ................................... SUCCESS [  0.000 s]
[INFO] Dinky : App 1.18 ................................... SUCCESS [  0.002 s]
[INFO] Dinky : App 1.19 ................................... SUCCESS [  0.001 s]
[INFO] Dinky : Admin ...................................... SUCCESS [  0.001 s]
[INFO] Dinky : Assembly ................................... SUCCESS [  0.000 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  20.686 s
[INFO] Finished at: 2024-06-13T20:50:47+08:00
[INFO] ------------------------------------------------------------------------
Zzm0809 commented 3 months ago

需要执行下代码格式化

我本地是可以的。。

> ./mvnw -T 4C -B --no-snapshot-updates clean spotless:check -P flink-all
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Dinky : Real-time Computing Platform                               [pom]
[INFO] Dinky : Flink                                                      [pom]
[INFO] Dinky : Flink 1.14                                                 [jar]
[INFO] Dinky : Flink 1.15                                                 [jar]
[INFO] Dinky : Flink 1.16                                                 [jar]
[INFO] Dinky : Flink 1.17                                                 [jar]
[INFO] Dinky : Flink 1.18                                                 [jar]
[INFO] Dinky : Flink 1.19                                                 [jar]
[INFO] Dinky : Common                                                     [jar]
[INFO] Dinky : Client                                                     [pom]
[INFO] Dinky : Client : Hadoop                                            [jar]
[INFO] Dinky : Client : Base                                              [jar]
[INFO] Dinky : Client 1.14                                                [jar]
[INFO] Dinky : Client 1.15                                                [jar]
[INFO] Dinky : Client 1.16                                                [jar]
[INFO] Dinky : Client 1.17                                                [jar]
[INFO] Dinky : Client 1.18                                                [jar]
[INFO] Dinky : Client 1.19                                                [jar]
[INFO] Dinky : CDC                                                        [pom]
[INFO] Dinky : CDC: Core                                                  [jar]
[INFO] Dinky : Catalog                                                    [pom]
[INFO] Dinky : Catalog : Mysql                                            [pom]
[INFO] Dinky : Catalog : Mysql 1.14                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.15                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.16                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.17                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.18                                       [jar]
[INFO] Dinky : Catalog : Mysql 1.19                                       [jar]
[INFO] Dinky : Gateway                                                    [jar]
[INFO] Dinky : Function                                                   [jar]
[INFO] Dinky : Metadata                                                   [pom]
......
[INFO] --- spotless-maven-plugin:2.27.1:apply (default-cli) @ dinky-assembly ---
[INFO] Spotless apply skipped
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Dinky : Real-time Computing Platform 1.1.0-SNAPSHOT:
[INFO] 
[INFO] Dinky : Real-time Computing Platform ............... SUCCESS [  0.077 s]
[INFO] Dinky : Flink ...................................... SUCCESS [  0.002 s]
[INFO] Dinky : Flink 1.14 ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Flink 1.15 ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Flink 1.16 ................................. SUCCESS [  0.002 s]
[INFO] Dinky : Flink 1.17 ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Flink 1.18 ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Flink 1.19 ................................. SUCCESS [  0.002 s]
[INFO] Dinky : Common ..................................... SUCCESS [  0.002 s]
[INFO] Dinky : Client ..................................... SUCCESS [  0.003 s]
[INFO] Dinky : Client : Hadoop ............................ SUCCESS [  0.001 s]
[INFO] Dinky : Client : Base .............................. SUCCESS [  0.000 s]
[INFO] Dinky : Client 1.14 ................................ SUCCESS [  0.001 s]
[INFO] Dinky : Client 1.15 ................................ SUCCESS [  0.002 s]
[INFO] Dinky : Client 1.16 ................................ SUCCESS [  0.001 s]
[INFO] Dinky : Client 1.17 ................................ SUCCESS [  0.002 s]
[INFO] Dinky : Client 1.18 ................................ SUCCESS [  0.002 s]
[INFO] Dinky : Client 1.19 ................................ SUCCESS [  0.001 s]
[INFO] Dinky : CDC ........................................ SUCCESS [  0.001 s]
[INFO] Dinky : CDC: Core .................................. SUCCESS [  0.000 s]
[INFO] Dinky : CDC: Plus .................................. SUCCESS [  0.001 s]
[INFO] Dinky : Catalog .................................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql ............................ SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.14 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.15 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.16 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.17 ....................... SUCCESS [  0.000 s]
[INFO] Dinky : Catalog : Mysql 1.18 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Catalog : Mysql 1.19 ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Gateway .................................... SUCCESS [  0.002 s]
[INFO] Dinky : Function ................................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata ................................... SUCCESS [  0.000 s]
[INFO] Dinky : Metadata : Base ............................ SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Mysql ........................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Oracle .......................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Clickhouse ...................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Postgresql ...................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Sqlserver ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Doris ........................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Phoenix ......................... SUCCESS [  0.000 s]
[INFO] Dinky : Metadata : Hive ............................ SUCCESS [  0.000 s]
[INFO] Dinky : Metadata : Starrocks ....................... SUCCESS [  0.001 s]
[INFO] Dinky : Metadata : Presto .......................... SUCCESS [  0.001 s]
[INFO] Dinky : Connectors ................................. SUCCESS [  0.001 s]
[INFO] Dinky : Connector : JDBC 1.14 ...................... SUCCESS [  0.001 s]
[INFO] Dinky : Extends .................................... SUCCESS [  0.000 s]
[INFO] Dinky : Alert ...................................... SUCCESS [  0.001 s]
[INFO] Dinky : Alter : Base ............................... SUCCESS [  0.001 s]
[INFO] Dinky : Alter : DingTalk ........................... SUCCESS [  0.000 s]
[INFO] Dinky : Alter : Wechat ............................. SUCCESS [  0.001 s]
[INFO] Dinky : Alter : FeiShu ............................. SUCCESS [  0.001 s]
[INFO] Dinky : Alter : Email .............................. SUCCESS [  0.001 s]
[INFO] Dinky : Alter : SMS ................................ SUCCESS [  0.000 s]
[INFO] Dinky : Alter : Http ............................... SUCCESS [  0.001 s]
[INFO] Dinky : Daemon ..................................... SUCCESS [  0.001 s]
[INFO] Dinky : Scheduler .................................. SUCCESS [  0.001 s]
[INFO] Dinky : Core ....................................... SUCCESS [  0.001 s]
[INFO] Dinky : App ........................................ SUCCESS [  0.002 s]
[INFO] Dinky : App : Base ................................. SUCCESS [  0.001 s]
[INFO] Dinky : App 1.14 ................................... SUCCESS [  0.001 s]
[INFO] Dinky : App 1.15 ................................... SUCCESS [  0.001 s]
[INFO] Dinky : App 1.16 ................................... SUCCESS [  0.001 s]
[INFO] Dinky : App 1.17 ................................... SUCCESS [  0.000 s]
[INFO] Dinky : App 1.18 ................................... SUCCESS [  0.002 s]
[INFO] Dinky : App 1.19 ................................... SUCCESS [  0.001 s]
[INFO] Dinky : Admin ...................................... SUCCESS [  0.001 s]
[INFO] Dinky : Assembly ................................... SUCCESS [  0.000 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  20.686 s
[INFO] Finished at: 2024-06-13T20:50:47+08:00
[INFO] ------------------------------------------------------------------------

代码格式化只针对已变更的文件做检查,由于你本地变更的文件已经提交, 所以无法重新进行格式化,你需要将存在格式化问题的文件随便修改一处,出现在变更列表内,然后运行 mvn spotless:apply

查看 CI 中出现问题链接: https://github.com/DataLinkDC/dinky/actions/runs/9500000123/job/26212058334?pr=3592

或者你可以根据 dinky 官方文档中的开发者指南配置服务端代码格式化 后期提交 pr 即可自动执行代码格式化,无需本地手动执行, 直达链接: https://www.dinky.org.cn/docs/next/developer_guide/contribution/code_format

soulmz commented 3 months ago

代码格式化只针对已变更的文件做检查,由于你本地变更的文件已经提交, 所以无法重新进行格式化,你需要将存在格式化问题的文件随便修改一处,出现在变更列表内,然后运行 mvn spotless:apply

查看 CI 中出现问题链接: https://github.com/DataLinkDC/dinky/actions/runs/9500000123/job/26212058334?pr=3592

或者你可以根据 dinky 官方文档中的开发者指南配置服务端代码格式化 后期提交 pr 即可自动执行代码格式化,无需本地手动执行, 直达链接: https://www.dinky.org.cn/docs/next/developer_guide/contribution/code_format

好的，我 fork 的仓库已经配置好，代码的风格已经可以了。

image