tikv scale test case takes 8h20m to complete sync task, is it reasonable? - Githubissues

pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)

Apache License 2.0

430 stars 286 forks source link

tikv scale test case takes 8h20m to complete sync task, is it reasonable? #3489

Open Tammyxia opened 3 years ago

Tammyxia commented 3 years ago

Before asking a question, make sure you have

[X] Searched existing Stack Overflow questions
[X] Googled your question
[X] Searched open and closed GitHub issues
[X] Read the documentation

What is your question?

Testbed: 3 cdc(16GB, 8C), 3 tikv (16GB, 8C), 1 changefeed, downstream is mysql8
Workload is tpcc 100 warehouse: go-tpc tpcc -- db workload --warehouses 100 -T 100 --host upstream-tidb.cdc-testbed--tps-450022-1-692 --port 4000 --parts 1 prepare; /go-tpc tpcc -- db workload --warehouses 100 -T 100 --host upstream-tidb.cdc-testbed--tps-450022-1-692 --port 4000 --parts 1 --time 1h0m0s run
Start to scale-out tikv pod to 7.
After 15minutes, scale-in tikv pod to 3.
Check sync task status, it takes 8h20m to complete sync data.
In this progress, one capture(not owner) restart once.

upstream tidb:

liuzix commented 3 years ago

Preliminary investigation shows no abnormalities other than:

node-1 restarted unexpectedly before the upstream workload finished, but it successfully restarted and continued to work.
node-1 writes to the downstream extremely slowly, but was able to finish writing all the data eventually.