search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.9k stars 570 forks source link

Created a mysql-cdc source, but did not configure schema history to kafka topic, causing RisingWave to hang up #18734

Open cumtcwf opened 2 weeks ago

cumtcwf commented 2 weeks ago

Describe the bug

  1. Created a mysql-cdc source, but did not configure schema history to kafka topic, causing RisingWave to hang up
  2. Created a risingwave table (t_test_risingwave) and mapped it to a table in the mysql-cdc source
  3. Cerated a mysql jdbc sink t_sink_risingwave from this table (t_test_risingwave)
  4. Modify the metadata of the MySQL table associated with the sink (change the type of a field from varchar to timestamp)
  5. Inserting new data into tables in MySQL will result in an error, causing RisingWave to crash

Error message/log

2024-09-26T06:55:32.642280809Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2239 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642287551Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2235 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642310586Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2234 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642300676Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2233 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642324382Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2241 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642317783Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2133 error=Executor error: exchange channel from local upstream actor 2162 closed unexpectedly
2024-09-26T06:55:32.642314431Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2242 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642355798Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2237 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642357431Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2230 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642440875Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2238 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.643181134Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2229 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.643444446Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2228 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.643458218Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2236 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.643512081Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2253 error=Executor error: exchange channel from local upstream actor 2138 closed unexpectedly
2024-09-26T06:55:32.643615174Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2232 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly

java.lang.RuntimeException: Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=RW_CDC_222'
        at io.debezium.pipeline.JmxUtils.registerMXBean(JmxUtils.java:68)
        at io.debezium.metrics.Metrics.register(Metrics.java:65)
        at io.debezium.relational.history.SchemaHistoryMetrics.started(SchemaHistoryMetrics.java:102)
        at io.debezium.relational.history.AbstractSchemaHistory.start(AbstractSchemaHistory.java:69)
        at io.debezium.relational.HistorizedRelationalDatabaseSchema.<init>(HistorizedRelationalDatabaseSchema.java:50)
        at io.debezium.connector.mysql.MySqlDatabaseSchema.<init>(MySqlDatabaseSchema.java:82)
        at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:98)
        at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:142)
        at io.debezium.embedded.EmbeddedEngine.startSourceTask(EmbeddedEngine.java:918)
        at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:737)
        at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:229)
        at com.risingwave.connector.source.core.DbzCdcEngine.run(DbzCdcEngine.java:66)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.InterruptedException: sleep interrupted
        at java.base/java.lang.Thread.sleep(Native Method)
        at io.debezium.util.Metronome$1.pause(Metronome.java:57)
        at io.debezium.pipeline.JmxUtils.registerMXBean(JmxUtils.java:59)
        ... 14 more

Caused by these errors (recent errors listed first):
  1: gRPC request to meta service failed: Internal error
  2: The cluster is recovering
  3: get error from control stream: worker node 2, gRPC request to stream service failed: Internal error: failed to collect barrier for epoch [7215546658455552]: Actor 2375 exited unexpectedly: Executor error: Sink error: Remote sink error: INTERNAL: Error when exec 22001, message Data truncation: Incorrect datetime value: 'cccc' for column 'column2' at row 1;

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

StrikeW commented 14 hours ago

Hi @cumtcwf , you can see in the error report Sink error: Remote sink error: INTERNAL: Error when exec 22001, message Data truncation: Incorrect datetime value: 'cccc' for column 'column2' at row 1;, since you changed the data type of table in mysql sink from varchar to timestamp you need to recreate the sink.