search

DataLinkDC / dinky

Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.
http://www.dinky.org.cn
Apache License 2.0
2.92k stars 1.07k forks source link

[Bug] [CDC Source] mysql cdc source just create tables but no data synch #3585

Closed sin70611 closed 2 weeks ago

sin70611 commented 3 weeks ago

Search before asking

What happened

MySQL cdcsource auto created two tables configed in the script，but no data synched from the mysql-cdc source； and no error log find in flink log and dinky log; and the overview of flink is normal as the img at the end

environment flink 1.17.2 dinky 1.0.2 flink cdc 3.0.1

sql script EXECUTE CDCSOURCE cdc_mysql WITH ( 'connector' = 'mysql-cdc', 'hostname' = '10.1.101.207', 'port' = '3306', 'username' = 'root', 'password' = 'yto123456', 'checkpoint' = '3000', 'parallelism' = '1', 'scan.startup.mode' = 'init', 'table-name' = 'cls_test.aircraft,cls_test.aircraftballast', 'sink.connector' = 'jdbc', 'sink.url' = 'jdbc:mysql://10.1.101.239:3306/test?characterEncoding=utf-8&useSSL=false', 'sink.username' = 'root', 'sink.password' = '0hS0}i$Juw0a', 'sink.sink.db' = 'test', 'sink.table.prefix' = 'pz', 'sink.table.lower' = 'true', 'sink.table-name' = '#{tableName}', 'sink.driver' = 'com.mysql.jdbc.Driver', 'sink.sink.buffer-flush.interval' = '2s', 'sink.sink.buffer-flush.max-rows' = '100', 'sink.sink.max-retries' = '5', 'sink.auto.create' = 'true', )

dinky log:

2024-06-11 09:32:36,532 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table cls_test.aircraft, the chunk size is 8096 from null 2024-06-11 09:32:36,788 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - ChunkSplitter has split 0 chunks for table cls_test.aircraft 2024-06-11 09:32:36,797 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table cls_test.aircraft into 1 chunks, time cost: 1622ms. 2024-06-11 09:32:36,797 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table cls_test.aircraft_ballast into chunks... 2024-06-11 09:32:36,832 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table cls_test.aircraft_ballast into 1 chunks, time cost: 35ms. 2024-06-11 09:32:36,931 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) (494abf615f98f75f465d0a01dd6c1028_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from SCHEDULED to DEPLOYING. 2024-06-11 09:32:36,934 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) (attempt #0) with attempt id 494abf615f98f75f465d0a01dd6c1028_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to localhost:45675-f7ba0b @ localhost (dataPort=43743) with allocation id 560aae724322a1a8d52361821a658514 2024-06-11 09:32:36,949 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[3] -> Sink: pz_aircraft[3] (1/1) (494abf615f98f75f465d0a01dd6c1028_1709cef93df857de2713dca78c0c9ff7_0_0) switched from SCHEDULED to DEPLOYING. 2024-06-11 09:32:36,949 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying SinkMaterializer[3] -> Sink: pz_aircraft[3] (1/1) (attempt #0) with attempt id 494abf615f98f75f465d0a01dd6c1028_1709cef93df857de2713dca78c0c9ff7_0_0 and vertex id 1709cef93df857de2713dca78c0c9ff7_0 to localhost:45675-f7ba0b @ localhost (dataPort=43743) with allocation id 560aae724322a1a8d52361821a658514 2024-06-11 09:32:36,970 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[6] -> Sink: pz_aircraft_ballast[6] (1/1) (494abf615f98f75f465d0a01dd6c1028_c3affa7051e234100feb8a4f53de3302_0_0) switched from SCHEDULED to DEPLOYING. 2024-06-11 09:32:36,970 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying SinkMaterializer[6] -> Sink: pz_aircraft_ballast[6] (1/1) (attempt #0) with attempt id 494abf615f98f75f465d0a01dd6c1028_c3affa7051e234100feb8a4f53de3302_0_0 and vertex id c3affa7051e234100feb8a4f53de3302_0 to localhost:45675-f7ba0b @ localhost (dataPort=43743) with allocation id 560aae724322a1a8d52361821a658514 2024-06-11 09:32:37,922 INFO org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger checkpoint for job 7f621cbd01827d1b77ce2b5185201ad1 since Checkpoint triggering task Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) of job 7f621cbd01827d1b77ce2b5185201ad1 is not being executed at the moment. Aborting checkpoint. Failure reason: Not all required tasks are currently running.. 2024-06-11 09:32:37,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[6] -> Sink: pz_aircraft_ballast[6] (1/1) (494abf615f98f75f465d0a01dd6c1028_c3affa7051e234100feb8a4f53de3302_0_0) switched from DEPLOYING to INITIALIZING. 2024-06-11 09:32:37,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[3] -> Sink: pz_aircraft[3] (1/1) (494abf615f98f75f465d0a01dd6c1028_1709cef93df857de2713dca78c0c9ff7_0_0) switched from DEPLOYING to INITIALIZING. 2024-06-11 09:32:38,065 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) (494abf615f98f75f465d0a01dd6c1028_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to INITIALIZING. 2024-06-11 09:32:40,264 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[6] -> Sink: pz_aircraft_ballast[6] (1/1) (494abf615f98f75f465d0a01dd6c1028_c3affa7051e234100feb8a4f53de3302_0_0) switched from INITIALIZING to RUNNING. 2024-06-11 09:32:40,343 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[3] -> Sink: pz_aircraft[3] (1/1) (494abf615f98f75f465d0a01dd6c1028_1709cef93df857de2713dca78c0c9ff7_0_0) switched from INITIALIZING to RUNNING. 2024-06-11 09:32:40,652 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: MySQL CDC Source registering reader for parallel task 0 (#0) @ localhost 2024-06-11 09:32:40,658 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: MySQL CDC Source received split request from parallel task 0 (#0) 2024-06-11 09:32:40,677 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) (494abf615f98f75f465d0a01dd6c1028_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from INITIALIZING to RUNNING. 2024-06-11 09:32:40,718 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator assigns split MySqlSnapshotSplit{tableId=cls_test.aircraft, splitId='cls_test.aircraft:0', splitKeyType=[id CHAR(32) NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} to subtask 0 2024-06-11 09:32:40,908 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 1 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1718069560897 for job 7f621cbd01827d1b77ce2b5185201ad1. 2024-06-11 09:32:41,010 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 1 for job 7f621cbd01827d1b77ce2b5185201ad1 (56901 bytes, checkpointDuration=111 ms, finalizationTime=2 ms). 2024-06-11 09:32:41,010 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 1 as completed for source Source: MySQL CDC Source. 2024-06-11 09:32:43,898 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 2 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1718069563897 for job 7f621cbd01827d1b77ce2b5185201ad1. 2024-06-11 09:32:44,176 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 2 for job 7f621cbd01827d1b77ce2b5185201ad1 (57011 bytes, checkpointDuration=279 ms, finalizationTime=0 ms). 2024-06-11 09:32:44,187 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator under INITIAL_ASSIGNING receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={cls_test.aircraft:0={ts_sec=0, file=binlog.000028, pos=110773642, kind=SPECIFIC, gtids=, row=0, event=0}}} from subtask 0. 2024-06-11 09:32:44,201 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: MySQL CDC Source received split request from parallel task 0 (#0) 2024-06-11 09:32:44,202 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator assigns split MySqlSnapshotSplit{tableId=cls_test.aircraft_ballast, splitId='cls_test.aircraft_ballast:0', splitKeyType=[id CHAR(32) NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} to subtask 0 2024-06-11 09:32:44,202 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 2 as completed for source Source: MySQL CDC Source. 2024-06-11 09:32:44,784 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator under INITIAL_ASSIGNING receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={cls_test.aircraft_ballast:0={ts_sec=0, file=binlog.000028, pos=110773642, kind=SPECIFIC, gtids=, row=0, event=0}}} from subtask 0. 2024-06-11 09:32:44,795 INFO com.ververica.cdc.connectors.mysql.source.assigners.AssignerStatus [] - Assigner status changes from INITIAL_ASSIGNING to INITIAL_ASSIGNING_FINISHED 2024-06-11 09:32:44,795 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Snapshot split assigner received all splits finished and the job parallelism is 1, snapshot split assigner is turn into finished status. 2024-06-11 09:32:44,795 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: MySQL CDC Source received split request from parallel task 0 (#0) 2024-06-11 09:32:44,875 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator assigns split MySqlBinlogSplit{splitId='binlog-split', offset={ts_sec=0, file=binlog.000028, pos=110773642, kind=SPECIFIC, gtids=, row=0, event=0}, endOffset={ts_sec=0, file=, pos=-9223372036854775808, kind=NON_STOPPING, row=0, event=0}, isSuspended=false} to subtask 0 2024-06-11 09:32:45,345 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator receives notice from subtask 0 for the binlog split assignment. 2024-06-11 09:32:46,898 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 3 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1718069566897 for job 7f621cbd01827d1b77ce2b5185201ad1.

flink log

2024-06-11 09:32:36,532 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table cls_test.aircraft, the chunk size is 8096 from null 2024-06-11 09:32:36,788 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - ChunkSplitter has split 0 chunks for table cls_test.aircraft 2024-06-11 09:32:36,797 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table cls_test.aircraft into 1 chunks, time cost: 1622ms. 2024-06-11 09:32:36,797 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table cls_test.aircraft_ballast into chunks... 2024-06-11 09:32:36,832 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table cls_test.aircraft_ballast into 1 chunks, time cost: 35ms. 2024-06-11 09:32:36,931 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) (494abf615f98f75f465d0a01dd6c1028_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from SCHEDULED to DEPLOYING. 2024-06-11 09:32:36,934 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) (attempt #0) with attempt id 494abf615f98f75f465d0a01dd6c1028_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to localhost:45675-f7ba0b @ localhost (dataPort=43743) with allocation id 560aae724322a1a8d52361821a658514 2024-06-11 09:32:36,949 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[3] -> Sink: pz_aircraft[3] (1/1) (494abf615f98f75f465d0a01dd6c1028_1709cef93df857de2713dca78c0c9ff7_0_0) switched from SCHEDULED to DEPLOYING. 2024-06-11 09:32:36,949 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying SinkMaterializer[3] -> Sink: pz_aircraft[3] (1/1) (attempt #0) with attempt id 494abf615f98f75f465d0a01dd6c1028_1709cef93df857de2713dca78c0c9ff7_0_0 and vertex id 1709cef93df857de2713dca78c0c9ff7_0 to localhost:45675-f7ba0b @ localhost (dataPort=43743) with allocation id 560aae724322a1a8d52361821a658514 2024-06-11 09:32:36,970 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[6] -> Sink: pz_aircraft_ballast[6] (1/1) (494abf615f98f75f465d0a01dd6c1028_c3affa7051e234100feb8a4f53de3302_0_0) switched from SCHEDULED to DEPLOYING. 2024-06-11 09:32:36,970 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying SinkMaterializer[6] -> Sink: pz_aircraft_ballast[6] (1/1) (attempt #0) with attempt id 494abf615f98f75f465d0a01dd6c1028_c3affa7051e234100feb8a4f53de3302_0_0 and vertex id c3affa7051e234100feb8a4f53de3302_0 to localhost:45675-f7ba0b @ localhost (dataPort=43743) with allocation id 560aae724322a1a8d52361821a658514 2024-06-11 09:32:37,922 INFO org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger checkpoint for job 7f621cbd01827d1b77ce2b5185201ad1 since Checkpoint triggering task Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) of job 7f621cbd01827d1b77ce2b5185201ad1 is not being executed at the moment. Aborting checkpoint. Failure reason: Not all required tasks are currently running.. 2024-06-11 09:32:37,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[6] -> Sink: pz_aircraft_ballast[6] (1/1) (494abf615f98f75f465d0a01dd6c1028_c3affa7051e234100feb8a4f53de3302_0_0) switched from DEPLOYING to INITIALIZING. 2024-06-11 09:32:37,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[3] -> Sink: pz_aircraft[3] (1/1) (494abf615f98f75f465d0a01dd6c1028_1709cef93df857de2713dca78c0c9ff7_0_0) switched from DEPLOYING to INITIALIZING. 2024-06-11 09:32:38,065 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) (494abf615f98f75f465d0a01dd6c1028_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to INITIALIZING. 2024-06-11 09:32:40,264 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[6] -> Sink: pz_aircraft_ballast[6] (1/1) (494abf615f98f75f465d0a01dd6c1028_c3affa7051e234100feb8a4f53de3302_0_0) switched from INITIALIZING to RUNNING. 2024-06-11 09:32:40,343 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - SinkMaterializer[3] -> Sink: pz_aircraft[3] (1/1) (494abf615f98f75f465d0a01dd6c1028_1709cef93df857de2713dca78c0c9ff7_0_0) switched from INITIALIZING to RUNNING. 2024-06-11 09:32:40,652 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: MySQL CDC Source registering reader for parallel task 0 (#0) @ localhost 2024-06-11 09:32:40,658 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: MySQL CDC Source received split request from parallel task 0 (#0) 2024-06-11 09:32:40,677 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MySQL CDC Source -> PartitionByPrimarykey -> Process -> (Flat Map -> anonymous_datastream_source$1[1] -> Calc[2] -> ConstraintEnforcer[3], Flat Map -> anonymous_datastream_source$2[4] -> Calc[5] -> ConstraintEnforcer[6]) (1/1) (494abf615f98f75f465d0a01dd6c1028_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from INITIALIZING to RUNNING. 2024-06-11 09:32:40,718 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator assigns split MySqlSnapshotSplit{tableId=cls_test.aircraft, splitId='cls_test.aircraft:0', splitKeyType=[id CHAR(32) NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} to subtask 0 2024-06-11 09:32:40,908 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 1 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1718069560897 for job 7f621cbd01827d1b77ce2b5185201ad1. 2024-06-11 09:32:41,010 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 1 for job 7f621cbd01827d1b77ce2b5185201ad1 (56901 bytes, checkpointDuration=111 ms, finalizationTime=2 ms). 2024-06-11 09:32:41,010 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 1 as completed for source Source: MySQL CDC Source. 2024-06-11 09:32:43,898 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 2 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1718069563897 for job 7f621cbd01827d1b77ce2b5185201ad1. 2024-06-11 09:32:44,176 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 2 for job 7f621cbd01827d1b77ce2b5185201ad1 (57011 bytes, checkpointDuration=279 ms, finalizationTime=0 ms). 2024-06-11 09:32:44,187 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator under INITIAL_ASSIGNING receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={cls_test.aircraft:0={ts_sec=0, file=binlog.000028, pos=110773642, kind=SPECIFIC, gtids=, row=0, event=0}}} from subtask 0. 2024-06-11 09:32:44,201 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: MySQL CDC Source received split request from parallel task 0 (#0) 2024-06-11 09:32:44,202 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator assigns split MySqlSnapshotSplit{tableId=cls_test.aircraft_ballast, splitId='cls_test.aircraft_ballast:0', splitKeyType=[id CHAR(32) NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} to subtask 0 2024-06-11 09:32:44,202 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 2 as completed for source Source: MySQL CDC Source. 2024-06-11 09:32:44,784 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator under INITIAL_ASSIGNING receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={cls_test.aircraft_ballast:0={ts_sec=0, file=binlog.000028, pos=110773642, kind=SPECIFIC, gtids=, row=0, event=0}}} from subtask 0. 2024-06-11 09:32:44,795 INFO com.ververica.cdc.connectors.mysql.source.assigners.AssignerStatus [] - Assigner status changes from INITIAL_ASSIGNING to INITIAL_ASSIGNING_FINISHED 2024-06-11 09:32:44,795 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Snapshot split assigner received all splits finished and the job parallelism is 1, snapshot split assigner is turn into finished status. 2024-06-11 09:32:44,795 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: MySQL CDC Source received split request from parallel task 0 (#0) 2024-06-11 09:32:44,875 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator assigns split MySqlBinlogSplit{splitId='binlog-split', offset={ts_sec=0, file=binlog.000028, pos=110773642, kind=SPECIFIC, gtids=, row=0, event=0}, endOffset={ts_sec=0, file=, pos=-9223372036854775808, kind=NON_STOPPING, row=0, event=0}, isSuspended=false} to subtask 0 2024-06-11 09:32:45,345 INFO com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator [] - The enumerator receives notice from subtask 0 for the binlog split assignment. 2024-06-11 09:32:46,898 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 3 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1718069566897 for job 7f621cbd01827d1b77ce2b5185201ad1.

image

What you expected to happen

get data synched from source to sink

How to reproduce

just execute the script in the environment above

Anything else

No response

Version

1.0.2

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 3 weeks ago

Hello @sin70611, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @sin70611, 这个 issue 是关于 CDC/CDCSOURCE 的，所以我把它分配给了 @aiwenmo。如有任何问题，可以评论回复。

Zzm0809 commented 3 weeks ago

'scan.startup.mode' = 'init', 是什么鬼??? 请先去查看 cdc 官网关于连接器的参数设置及其值

另外: 请在提问题时 注意将主机 ip 端口,账号密码等信息脱敏

github-actions[bot] commented 3 weeks ago

Hello @sin70611, thank you for your feedback, but this issue will not be fixed. You can search for keywords in the Issue list to view it. Issues are marked `Wont Fix`

你好 @sin70611, 感谢你的反馈, 但是此问题不会被修复，你可以在 Issue 列表中搜索关键字来查看, Issues 标注为 `Wont Fix` 。

sin70611 commented 1 week ago

tinyint类型字段 metadata_table_property 会定义成int； tinyint（1）类型字段 metadata_table_property 会定义成boolean，造成数据表无法同步。