[Bug] [Module Name] CDCSOURCE task was submitted successfully, but the data could not be synchronized.

cuanxiaopo commented 7 months ago

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

dink1.0.1, flink1.17。在测试整库同步时，任务提交成功，但是数据没有同步过去，全量数据，增量数据都无法同步过去，也没有报错。 EXECUTE CDCSOURCE cdc_sms WITH ( 'connector' = 'mysql-cdc', 'hostname' = 'xxxxx', 'port' = '8527', 'username' = 'root', 'password' = '123456', 'checkpoint' = '3000', 'scan.startup.mode' = 'initial', 'parallelism' = '1', 'table-name' = 'sms.sms_type,sms.smsreceive', 'source.server-time-zone' = '+08:00', 'sink.connector' = 'jdbc', 'sink.url' = 'jdbc:mysql://xxxxxx:3306/sms?characterEncoding=utf-8&useSSL=false', 'sink.username' = 'root', 'sink.password' = '123456', 'sink.sink.db' = 'sms', 'sink.table.prefix' = 'ods', 'sink.table.lower' = 'true', 'sink.table-name' = '#{tableName}', 'sink.driver' = 'com.mysql.jdbc.Driver', 'sink.sink.buffer-flush.interval' = '2s', 'sink.sink.buffer-flush.max-rows' = '100', 'sink.sink.max-retries' = '5', 'sink.auto.create' = 'true' )

What you expected to happen

全量数据，增量数据正常同步，且实现表结构同步变更

How to reproduce

1

Anything else

No response

Version

1.0.0

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

github-actions[bot] commented 7 months ago

Hello @cuanxiaopo, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @cuanxiaopo, 这个 issue 是关于 CDC/CDCSOURCE 的，所以我把它分配给了 @aiwenmo。如有任何问题，可以评论回复。

cuanxiaopo commented 7 months ago

msyql到mysql的整库同步。

cuanxiaopo commented 7 months ago

我之前测试0.7.5也遇到过类似的问题，现在用1.0.1之后还是同样的问题。表可以创建成功，数据无法同步过来

cuanxiaopo commented 7 months ago

全量数据可以同步，增量数据无法同步

Zzm0809 commented 7 months ago

set checkpoint 参数

cuanxiaopo commented 7 months ago

set checkpoint 参数

全量同步语法中不是已经有 checkpoint设置了吗。在set下还是不行（SET 'execution.checkpointing.interval' = '3s';），增量数据仍然无法同步。

Zzm0809 commented 7 months ago

set checkpoint 参数

全量同步语法中不是已经有 checkpoint设置了吗。在set下还是不行（SET 'execution.checkpointing.interval' = '3s';），增量数据仍然无法同步。

无法复现

cuanxiaopo commented 7 months ago

set checkpoint 参数

全量同步语法中不是已经有 checkpoint设置了吗。在set下还是不行（SET 'execution.checkpointing.interval' = '3s';），增量数据仍然无法同步。

无法复现

1 mysql-mysql整库同步成功了，增量数据也可以了。（但是不能实现表结构变更的同步吗？） 2 昨天了解到某个测试库是polardb，并不是mysql。目前polardb-mysql单表可以，整库同步仍存在这个问题。

DataLinkDC / dinky