[Bug]: CDC table which 600 million rows to mysql failed

[heni02]

Is there an existing issue for the same bug?

• [X] I have checked the existing issues.

Branch Name

main

Commit ID

786f7bb

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

测试环境：3cn minio 测试现象：6亿行的表，mo同步到mysql，1个多小时下游mysql表还是0行 测试步骤：上游load数据前，创建同步任务，之后上游开始load数据

下游msyql表信息：

上游mo表信息：

同步命令： ./mo_cdc task create --task-name cdc_lineitem --source-uri="mysql://dump:111@10.222.6.6:6001" --sink-type="mysql" --sink-uri="mysql://dump:111@10.222.1.129:3306" --tables='testdb.lineitem:back_ac1_db.LINEITEM' --level="account" --account="sys"

上游mo load表耗时7min+

上游表ddl和load sql见复现步骤

Expected Behavior

No response

Steps to Reproduce

ddl：
CREATE TABLE LINEITEM(
L_ORDERKEY    BIGINT NOT NULL,
L_PARTKEY     INTEGER NOT NULL,
L_SUPPKEY     INTEGER NOT NULL,
L_LINENUMBER  INTEGER NOT NULL,
L_QUANTITY    DECIMAL(15,2) NOT NULL,
L_EXTENDEDPRICE  DECIMAL(15,2) NOT NULL,
L_DISCOUNT    DECIMAL(15,2) NOT NULL,
L_TAX         DECIMAL(15,2) NOT NULL,
L_RETURNFLAG  VARCHAR(1) NOT NULL,
L_LINESTATUS  VARCHAR(1) NOT NULL,
L_SHIPDATE    DATE NOT NULL,
L_COMMITDATE  DATE NOT NULL,
L_RECEIPTDATE DATE NOT NULL,
L_SHIPINSTRUCT CHAR(25) NOT NULL,
L_SHIPMODE     CHAR(10) NOT NULL,
L_COMMENT      VARCHAR(44) NOT NULL,
PRIMARY KEY (L_ORDERKEY, L_LINENUMBER)
);
load data url s3option {'endpoint'='http://minio.minio-mo.svc.cluster.local','access_key_id'='xxx','secret_access_key'='xxx','bucket'='mo-load-data', 'filepath'='tpch_100/lineitem.tbl','provider'='minio'} into table lineitem fields terminated
by '|' lines terminated by '\n' parallel 'true';

Additional information

No response

[heni02]

新一轮6亿同步数据测试大概15个小时左右才同步完成 另外同步期间cn会oom，不停发生重启现象

[heni02]

第三次6亿数据加载测试 commit：57be43198 测试结论：6亿数据同步完数据多了3万多行，内存从开始截止到现在最高12G，同步结束像是也没释放，同步完数据大概耗时8个小时左右

mem资源： http://10.222.6.1/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?orgId=1&var-datasource=prometheus&var-cluster=&var-namespace=mo-cdc-test&from=1728533570000&to=1728612770000 日志： http://10.222.6.1/explore?panes=%7B%22VCp%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-cdc-test%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221728532800000%22,%22to%22:%221728613080000%22%7D%7D%7D&schemaVersion=1&orgId=1

[ck89119]

同步失败的问题已经解决，性能问题测试中

[ck89119]

将snapshot类型change数据攒满max_packet_size再发送，待周一到测试环境测试

[ck89119]

测试环境硬件故障，测试中

[ck89119]

优化后大概有50%性能提升，具体数据更新在表格中了

[heni02]

6亿数据75G同步最新测试结果：同步数据一致，内存没有oom，同步耗时大概4个小时，同步速度5.33MB/s 测试环境：3cn （mem 25G，7core） commit:5300e8942c22438882fa8f7497cdbdcef7a418b9 同步耗时

https://shanghai.idc.matrixorigin.cn:30001/d/fe1m4fa3hc1kwa/frontend-metrics?orgId=1&var-interval=1m&var-namespace=mo-cdc-test&var-pod=All&from=1729764975398&to=1729788680814

[heni02]

confirm，closed