apache / flink-cdc

Flink CDC is a streaming data integration tool
https://nightlies.apache.org/flink/flink-cdc-docs-stable
Apache License 2.0
5.67k stars 1.92k forks source link

mysql-cdc CPU utilization100% #1580

Closed calm-projects closed 8 months ago

calm-projects commented 2 years ago

Flink mysql-cdc CPU usage is 100%

The CDC can read data, but the mysql Server monitor reports a large number of error logs, and the CPU usage is 100%, causing the mysql Server to be unavailable Aborted connection 38868 to db: 'unconnected' user: 'xxxx' host: 'xxx' (Failed on my_net_write()) Aborted connection 17042 to db: 'unconnected' user: 'xxx' host: 'xxx' (Got timeout reading communication packets)

Environment :

To Reproduce Steps to reproduce the behavior:

  1. The test data : About 50 million pieces of data,21:10 start cdc, 06:14 close cdc. Parallelism 1.
  2. The test code :CREATE TABLE userLogTmp ( id BIGINT, user_id BIGINT, route_url STRING, start_time TIMESTAMP, end_time TIMESTAMP, time BIGINT, PRIMARY KEY(id) NOT ENFORCED ) WITH ( 'connector' = 'mysql-cdc', 'hostname' = 'hadoop02', 'port' = '3306', 'username' = 'xxx', 'password' = 'xxx', 'database-name' = 'test', 'table-name' = 'userlog', 'connect.timeout' = '30s', 'connect.max-retries' = '2', 'connection.pool.size' = '5', 'scan.startup.mode' = 'initial', 'server-time-zone' = 'Asia/Shanghai', 'server-id' = '5452-5453' )
  3. The error : The CDC can read data, but the mysql Server monitor reports a large number of error logs, and the CPU usage is 100%, causing the mysql Server to be unavailable. Aborted connection 38868 to db: 'unconnected' user: 'xxxx' host: 'xxx' (Failed on my_net_write()) Aborted connection 17042 to db: 'unconnected' user: 'xxx' host: 'xxx' (Got timeout reading communication packets)

Additional Description aee88f99d06cdf69020845df32095af fa09a5b32a019a768f06babf19ea0b0

PatrickRen commented 8 months ago

Closing this issue because it was created before version 2.3.0 (2022-11-10). Please try the latest version of Flink CDC to see if the issue has been resolved. If the issue is still valid, kindly report it on Apache Jira under project Flink with component tag Flink CDC. Thank you!