search

pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
424 stars 282 forks source link

TiCDC OOM in the network loss frequently scenario #6221

Closed 3AceShowHand closed 4 months ago

3AceShowHand commented 2 years ago

What did you do?

Deploy 3 TiCDC, 8 CPU / 16Gi.

1 changefeed with MySQL Sink, replicating 100 tables, each 500000 rows, sysbench oltp_update_non_index

tableCount: 100 tableSize: 500000 testName: "oltp_update_non_index" threads: 100 duration: "2h"

What did you expect to see?

TiCDC should works normally, shouldn't out of memory.

What did you see instead?

TiCDC instance crashed since out of memory and meet CrashLoopBackOff

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

master

Upstream TiKV version (execute tikv-server --version):

master

TiCDC version (execute cdc version):

master
asddongmen commented 2 years ago

I also meet this problem. image image image

pprof.cdc.samples.cpu.001.pb.gz pprof.cdc.alloc_objects.alloc_space.inuse_objects.inuse_space.006.pb.gz pprof.cdc.alloc_objects.alloc_space.inuse_objects.inuse_space.007.pb.gz

asddongmen commented 4 months ago

Close it since it's stale.