br backup failed with error “rpc error: code = Unavailable desc = etcdserver: leader changed” after injection pd leader io delay

Lily2025 commented 2 hours ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run br backup 2、injection pd leader io delay

2. What did you expect to see? (Required)

br backup succeed

3. What did you see instead (Required)

br backup failed br.log.2024-10-31T04.27.51Z.zip

Detail BR log in /tmp/br.log.2024-10-31T04.27.51Z {level:warn,ts:2024-10-31T04:28:54.823024Z,logger:etcd-client,caller:v3@v3.5.12/retry_interceptor.go:62,msg:retrying of unary invoker failed,target:etcd-endpoints://0xc0022a08c0/tc-pd.ha-test-br-backup-tps-7643347-1-341:2379,attempt:0,error:rpc error: code = Unavailable desc = etcdserver: leader changed} {level:warn,ts:2024-10-31T04:28:54.966914Z,logger:etcd-client,caller:v3@v3.5.12/retry_interceptor.go:62,msg:retrying of unary invoker failed,target:etcd-endpoints://0xc0022a08c0/tc-pd.ha-test-br-backup-tps-7643347-1-341:237"

4. What is your TiDB version? (Required)

./br -V Release Version: v8.5.0-alpha-19-g49c3eba4b0 Git Commit Hash: 49c3eba4b061a3098e0095dfd7803b955948ce94 Git Branch: HEAD Go Version: go1.23.2 UTC Build Time: 2024-10-30 15:51:01 Race Enabled: false

Lily2025 commented 2 hours ago

/assign Leavrth

Lily2025 commented 2 hours ago

/severity major

Leavrth commented 22 minutes ago

[tikv:9001]PD server timeout: 
github.com/pingcap/errors.AddStack
    /root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/errors.go:178
github.com/pingcap/errors.(*Error).GenWithStackByArgs
    /root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/normalize.go:175
github.com/pingcap/tidb/pkg/store/driver/error.ToTiDBErr
    /workspace/source/tidb/pkg/store/driver/error/error.go:114
github.com/pingcap/tidb/pkg/store/driver/txn.(*tikvSnapshot).Iter
    /workspace/source/tidb/pkg/store/driver/txn/snapshot.go:72
github.com/pingcap/tidb/pkg/structure.(*TxStructure).IterateHash
    /workspace/source/tidb/pkg/structure/hash.go:226
github.com/pingcap/tidb/pkg/structure.(*TxStructure).HGetAll
    /workspace/source/tidb/pkg/structure/hash.go:157
github.com/pingcap/tidb/pkg/meta.(*Mutator).ListDatabases
    /workspace/source/tidb/pkg/meta/meta.go:1235
github.com/pingcap/tidb/br/pkg/backup.BuildBackupSchemas
    /workspace/source/tidb/br/pkg/backup/client.go:826
github.com/pingcap/tidb/br/pkg/backup.BuildBackupRangeAndInitSchema.func2
    /workspace/source/tidb/br/pkg/backup/client.go:812
github.com/pingcap/tidb/br/pkg/backup.(*Schemas).BackupSchemas
    /workspace/source/tidb/br/pkg/backup/schema.go:95
github.com/pingcap/tidb/br/pkg/task.RunBackup
    /workspace/source/tidb/br/pkg/task/backup.go:713
main.runBackupCommand
    /workspace/source/tidb/br/cmd/br/backup.go:56
main.newDBBackupCommand.func1
    /workspace/source/tidb/br/cmd/br/backup.go:163
github.com/spf13/cobra.(*Command).execute
    /root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
    /root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
    /root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
main.main
    /workspace/source/tidb/br/cmd/br/main.go:36
runtime.main
    /usr/local/go/src/runtime/proc.go:272
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1700

pingcap / tidb