JNSimba
commented
2 months ago You can check the configuration of jm and tm on flink web to see if the checkpoint and other information have been passed.
Open lliuql opened 6 months ago
JNSimba
commented
2 months ago You can check the configuration of jm and tm on flink web to see if the checkpoint and other information have been passed.
Search before asking
Version
java version "1.8.0_212" flink-1.17.1 hadoop3.3.4 下面三个jar都试过了 -rw-rw-r--. 1 liuqh liuqh 8452171 2月 23 17:28 flink-doris-connector-1.17-1.4.0.jar -rw-rw-r--. 1 liuqh liuqh 11915843 2月 23 17:28 flink-doris-connector-1.17-1.5.0.jar -rw-rw-r--. 1 liuqh liuqh 12193974 2月 23 17:28 flink-doris-connector-1.17-1.5.1.jar
What's Wrong?
在Standalone与Yarn Application模式下都能正常运行。但在Yarn Session模式下，数据同步存在问题。例如，当MySQL库中的test表有5条数据时，开启整库同步后，虽然成功创建了test表到Doris，但doris表数据为0条。随后对MySQL中的test表结构修改，表结构能够更新到Doris，但doris的test数据依然是0条
What You Expected?
成功同步数据
How to Reproduce?
`
1.5.0yarn-session没办法删除数据，但是可以同步表结构
bin/flink run \ -Dexecution.checkpointing.interval=10s \ -Dstate.checkpoints.dir=hdfs://hadoop102:8020/flink/checkpoints/report_sync \ -Dstate.savepoints.dir=hdfs://hadoop102:8020/flink/flink-savepoints/report_sync \ -Dparallelism.default=1 \ -c org.apache.doris.flink.tools.cdc.CdcTools \ job_jar/flink-doris-connector-1.17-1.5.0.jar \ mysql-sync-database \ --database test \ --mysql-conf \ server-id=5310-5313 \ --mysql-conf hostname=192.168.100.128 \ --mysql-conf port=3306 \ --mysql-conf username=root \ --mysql-conf password='123456' \ --mysql-conf database-name=test \ --sink-conf fenodes=192.168.88.120:8030 \ --sink-conf username=root \ --sink-conf password='Ryuu1234@' \ --sink-conf jdbc-url=jdbc:mysql://192.168.88.120:9030 \ --sink-conf sink.label-prefix=testx_sync2 \ --table-conf replication_num=1 \ --including-tables "test|test_tb01" `
Anything Else?
No response
Are you willing to submit PR?
Code of Conduct