CDC cloud: row lost when scale-out and scale-in

[Tammyxia]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

• Load data to upstream tidb: $ bin/go-ycsb run mysql -P workloads/betting -p operationcount=5000000 -p mysql.host=tidb.88d9efdd.50c5d2e.us-west-2.shared.aws.tidbcloud.com -p mysql.port=4000 --threads 200 -p dbnameprefix=testcc -p databaseproportions=1.0 -p unitnameprefix=unit1 -p unitscount=1 -p tablecount=100 -p mysql.password=12345678

• During data loading, scale-in then scale out cdc:

  1. scale-in cdc: $kubectl edit tc xxx -> modify ticdc: replica from 3 to 1, so this process cdc owner switched.
  2. scale-out cdc: $kubectl edit tc xxx -> modify ticdc: replica from 1 to 3

• Stop data loading, waiting for cdc sync task completing.

• Check if data consistency with sync-diff, config.toml: log-level = "info" chunk-size = 1000000 check-thread-count = 40 sample-percent = 100 use-checksum = true only-use-checksum = false use-checkpoint = true ignore-data-check = false ignore-struct-check = false fix-sql-file = "fix.sql"

[[check-tables]] schema = "testcc0" tables = ["~^"]

[[source-db]] host = "xxx" port = 4000 user = "root" password = "xxx" instance-id = "source-1"

[target-db] host = "xxx" port = 4000 user = "root" password = "xxx"

2. What did you expect to see?

• data consistency

3. What did you see instead?

• sync_diff has many logs about: target lack of data, find different row. For example, compare row count between upstream and downstream:

4. Versions of the cluster

   • Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     5. 0. 2

   • TiCDC version (execute cdc version):

     ["Welcome to Change Data Capture (CDC)"] [release-version=v5.0.0-dev] [git-hash=9300b05ceec2c4811198416a5d21e9f4910deddd] [git-branch=cloud-cdc-5.0] [utc-build-time="2021-06-26 14:16:57"] [go-version="go version go1.16.3 linux/amd64"] [failpoint-build=false]

[amyangfei]

Verify this issue after #2230 is fixed, since

• 2230 and this issue have same behaviors.

• This issue doesn't have more information for investigation