CDC cloud: workload "go-ycsb run" sync latency is too large(upstream write data 10m, sync task need 2hours to complete)

[Tammyxia]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

• Prepare workload to upstream tidb: go-ycsb load -P workloads/betting -p recordcount=500000 --threads 200 -p dbnameprefix=testcc -p databaseproportions=1.0 -p unitnameprefix=unit1 -p unitscount=1 -p tablecount=100 -p loadbatchsize=500

• After changefeed sync completing, load more data: go-ycsb run mysql -P workloads/betting -p operationcount=5000000 --threads 200 -p dbnameprefix=testcc -p databaseproportions=1.0 -p unitnameprefix=unit1 -p unitscount=1 -p tablecount=100 , this workload takes 10 minutes to finish.

• Check changefeed status and latency.

2. What did you expect to see?

• latency <= 5 minutes.

3. What did you see instead?

• Max latency is 2 hours.

4. Versions of the cluster

   • Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     5.0.2

   • TiCDC version (execute cdc version):

     ["Welcome to Change Data Capture (CDC)"] [release-version=v5.0.0-dev] [git-hash=9300b05ceec2c4811198416a5d21e9f4910deddd] [git-branch=cloud-cdc-5.0] [utc-build-time="2021-06-26 14:16:57"] [go-version="go version go1.16.3 linux/amd64"] [failpoint-build=false]

[amyangfei]

From downstream TiDB monitor we can observe the transaction duration is high.

[overvenus]

The root cause is TiCDC and downstream were deployed in different regions, the latency is high.