Data consistency when replicting tpcc workload to mysql sink

[fubinzh]

What did you do?

1. Upstream: TiDB cluster with 1 cdc node; CDC downstream: MySQL
2. Create mysql sink changefeed
3. Run tpcc workload

   /go-tpc tpcc --db workload --warehouses 1000 -T 32 --host upstream-tidb.cdc-testbed-tps-7593068-1-644 --port 4000 --parts 1 prepare --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062'
   /go-tpc tpcc --db workload --warehouses 1000 -T 32 --host upstream-tidb.cdc-testbed-tps-7593068-1-644 --port 4000 --parts 1 --time 5h0m0s run --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062'

4. Wait sync finished and do data consistency check between upstrem and downstream

What did you expect to see?

Data should be consistent

What did you see instead?

Data inconsistency, and cdc panic seen.

[2024/06/25 01:21:47.816 +00:00] [PANIC] [shared_region_worker.go:286] ["The CommitTs must be greater than the resolvedTs"] [EventType=COMMITTED] [CommitTs=450698541640712196] [resolvedTs=450698544943726600] [regionID=676] [stack="github.com/pingcap/tiflow/cdc/kv.handleEventEntry\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:286\ngithub.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).handleEventEntry\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:237\ngithub.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).processEvent\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:197\ngithub.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).run\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:164\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func1\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:346\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78"]

Versions of the cluster

Release Version: v8.2.0-alpha
Git Commit Hash: c001638fcf8cbe1790ca5d005a9a9464cfa53f84
Git Branch: heads/refs/tags/v8.2.0-alpha
UTC Build Time: 2024-06-21 11:36:49
Go Version: go version go1.21.10 linux/amd64
Failpoint Build: false

[fubinzh]

/severity critical