Redo log: test result is not meet the commitment "RTO<=10s"

Tammyxia commented 2 years ago

What did you do?

Create redo enabled changefeed
Run workload: /go-ycsb load mysql -P xxx -p operationcount=5000 -p autocommit=1 -p databasename=workload -p tablecount=10 -p mysql.host=upstream-tidb.cdc-testbedbtnlx -p mysql.port=4000 --threads=20
After workload finished, waiting for 10s to let cdc sync data
Stop changefed, then apply redo log
Check data between upstream and downstream

What did you expect to see?

Since redo log commit RTO < 10s, so when upstream workload finished for 10s, then let upstream crash, the expected result is cdc can sync all data in upstream to downstream.

What did you see instead?

Upstream has 10 tables, dowstream only has 2 tables.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)
/ # /cdc version
Release Version: v6.2.0-alpha
Git Commit Hash: fa41e47819b1038b808ce0e34ea386be3e95563b
Git Branch: heads/refs/tags/v6.2.0-alpha
UTC Build Time: 2022-08-09 13:08:24
Go Version: go version go1.18.5 linux/amd64
Failpoint Build: false

nongfushanquan commented 2 years ago

redo log will cache the events for 2s which may invoke the flow controlling , so set per-table-memory-quota to a decent value if the QPS is high. And also make sure the performance of disk isn't a bottleneck .

nongfushanquan commented 2 years ago

/close

ti-chi-bot commented 2 years ago

@nongfushanquan: Closing this issue.

In response to [this](https://github.com/pingcap/tiflow/issues/6699#issuecomment-1217424895): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

pingcap / tiflow