What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
What's the status of the TiDB cluster pods?
What did you do?
1.create volumebackup
What did you expect to see?
volumebackup success
What did you see instead?
volume failed
suspect that the pod was restarted for some reason, which caused the backup to be executed twice. The start time of the job is 8:41, the start time of the pod is 8:44, and the upload time of the backupmeta file is 8:42.
I0704 12:44:04.972263 9 backup.go:292] [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
I0704 12:44:04.972576 9 backup.go:292] [2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
I0704 12:44:04.977361 9 backup.go:292]
I0704 12:44:04.977393 9 backup.go:299] Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument
E0704 12:44:04.977497 9 manager.go:367] backup cluster backup-a/fed-backup-20230704-full-zm-dataplane-a data failed, err: cluster backup-a/fed-backup-20230704-full-zm-dataplane-a, wait pipe message failed, errMsg [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument
, err: exit status 1
I0704 12:44:04.990386 9 backup_status_updater.go:123] Backup: [backup-a/fed-backup-20230704-full-zm-dataplane-a] updated successfully
E0704 12:44:05.225639 9 backupsize.go:156] failed to get snapshots size 0, number of api request 0
W0704 12:44:05.225665 9 manager.go:324] Failed to calc volume snapshot backup size 0 bytes, ResourceNotFoundException: The snapshot 'snap-0a11162eaef40f5be' does not exist.
{
RespMetadata: {
StatusCode: 404,
RequestID: "56247a5c-9d56-4a50-ab97-6ffd8ccc0ddf"
},
Message: "The snapshot 'snap-0a11162eaef40f5be' does not exist.",
Reason: "SNAPSHOT_NOT_FOUND"
}
error: cluster backup-a/fed-backup-20230704-full-zm-dataplane-a, wait pipe message failed, errMsg [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument
, err: exit status 1
zhongmin@zhongmindeMacBook-P
Bug Report
What version of Kubernetes are you using?
What version of TiDB Operator are you using?
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
What's the status of the TiDB cluster pods?
What did you do? 1.create volumebackup
What did you expect to see?
What did you see instead?
I0704 12:44:04.972263 9 backup.go:292] [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] I0704 12:44:04.972576 9 backup.go:292] [2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] I0704 12:44:04.977361 9 backup.go:292] I0704 12:44:04.977393 9 backup.go:299] Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument E0704 12:44:04.977497 9 manager.go:367] backup cluster backup-a/fed-backup-20230704-full-zm-dataplane-a data failed, err: cluster backup-a/fed-backup-20230704-full-zm-dataplane-a, wait pipe message failed, errMsg [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument , err: exit status 1 I0704 12:44:04.990386 9 backup_status_updater.go:123] Backup: [backup-a/fed-backup-20230704-full-zm-dataplane-a] updated successfully E0704 12:44:05.225639 9 backupsize.go:156] failed to get snapshots size 0, number of api request 0 W0704 12:44:05.225665 9 manager.go:324] Failed to calc volume snapshot backup size 0 bytes, ResourceNotFoundException: The snapshot 'snap-0a11162eaef40f5be' does not exist. { RespMetadata: { StatusCode: 404, RequestID: "56247a5c-9d56-4a50-ab97-6ffd8ccc0ddf" }, Message: "The snapshot 'snap-0a11162eaef40f5be' does not exist.", Reason: "SNAPSHOT_NOT_FOUND" } error: cluster backup-a/fed-backup-20230704-full-zm-dataplane-a, wait pipe message failed, errMsg [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument , err: exit status 1 zhongmin@zhongmindeMacBook-P