pingcap / tiflash

The analytical engine for TiDB and TiDB Cloud. Try free: https://tidbcloud.com/free-trial
https://docs.pingcap.com/tidb/stable/tiflash-overview
Apache License 2.0
944 stars 410 forks source link

workload report “EstablishDisaggTask Failed4: Deadline Exceeded, e.what() = DB::Exception” after fault recover from s3 network_latency last for 10mins #7889

Open Lily2025 opened 1 year ago

Lily2025 commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run workload go-tpc ch run -D tpcc --host tc-tidb.endless-ha-test-tps-1830777-1-164 -P4000 --warehouses 2000 -T 32 --acThreads 1 --queries q22 --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062' --time 36000m --user root --password '' --interval '10s' 2、inject s3 network_latency(50ms) last for 10mins inject fault:2023-08-01 19:39:00 recover fault:2023-08-01 19:49:00

2. What did you expect to see? (Required)

no error code from workload

3. What did you see instead (Required)

workload report error after fault recover from 2023-08-01 19:53:00 execute run failed, err execute query q22 failed Error 1105: other error for mpp stream: Code: 0, e.displayText() = DB::Exception: EstablishDisaggTask Failed4: Deadline Exceeded, e.what() = DB::Exception,\r\n

4. What is your TiFlash version? (Required)

githash:759bb3e19d8b381952ab644a78948d8cc0bf28c0

Lily2025 commented 1 year ago

/assign JinheLin