search

risingwavelabs / risingwave

SQL stream processing, analytics, and management. We decouple storage and compute to offer efficient joins, instant failover, dynamic scaling, speedy bootstrapping, and concurrent query serving.
https://www.risingwave.com/slack
Apache License 2.0
6.64k stars 545 forks source link

citus-cdc: data synchronization will be slow when creating multiple sources #16193

Open xuefengze opened 3 months ago

xuefengze commented 3 months ago

test scripts can be found in https://github.com/risingwavelabs/risingwave/pull/16189

When testing locally, it takes some times to see data in citus_all_types. When testing in buildkite, although citus_all_types contains data during the data_check operation, the rows count is 2 instead of the expected 3.

https://buildkite.com/risingwavelabs/integration-tests/builds/734#018ebc7d-fd5f-42cb-bf94-331ed0c029d4

image
StrikeW commented 3 months ago

When testing locally, it takes some times to see data in citus_all_types.

Due to the conflict of JMX metric bean:

2024-04-08T02:06:36.109246969Z  WARN ThreadId(67) risingwave_connector_node: Unable to register metrics as an old set with the same name exists, retrying in PT5S (attempt 1 out of 12) thread="debezium-postgresconnector-RW_CDC_1-change-event-source-coordinator" class="io.debezium.pipeline.JmxUtils"
2024-04-08T02:06:36.208603076Z ERROR ThreadId(68) risingwave_connector_node: Failed to register metrics MBean, metrics will not be available thread="debezium-postgresconnector-RW_CDC_0-change-event-source-coordinator" class="io.debezium.pipeline.JmxUtils"

Need to take a deep look.