search

StarRocks / starrocks-connector-for-apache-flink

Apache License 2.0
192 stars 154 forks source link

Flink MySQL CDC writes to a Starrocks primary key model table, facing data synchronization latency issue #273

Closed nathan-szz closed 1 year ago

nathan-szz commented 1 year ago

I discovered that in a multi-table synchronization task, such as multiple 'insert into ...' statements, when testing data insertion into Starrocks, there is a delay of 5 minutes for one particular table. However, when performing a synchronization task for a single table alone, the delay returns to normal, at around 1 minute.

nathan-szz commented 1 year ago

Here is my configuration information: starrocks: 'sink.max-retries' = '10', 'sink.buffer-flush.interval-ms' = '30000', 'sink.properties.strip_outer_array' = 'true', 'sink.properties.format' = 'json', 'sink.properties.ignore_json_size' = 'true', 'sink.properties.connectTimeout' = '10000', 'sink.connect.timeout-ms' = '10000', 'sink.version'='V2'

flink：

execution.checkpointing.interval: 30000 state.checkpoints.num-retained: 20
execution.checkpointing.mode: EXACTLY_ONCE

banmoy commented 1 year ago

How did you measure the delay? Did the flink job have back pressure?

banmoy commented 1 year ago

What's the version of connector?

nathan-szz commented 1 year ago

What's the version of connector?

flink-connector-starrocks-1.2.7_flink-1.15

nathan-szz commented 1 year ago

I found reason what cause this bug. it’s not the connector .