search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.23k stars 498 forks source link

backup to be executed twice caused job failed [restore for 5143] #5788

Open ti-chi-bot opened 2 hours ago

ti-chi-bot commented 2 hours ago

Bug Report

What version of Kubernetes are you using?

What version of TiDB Operator are you using?

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

What's the status of the TiDB cluster pods?

What did you do? 1.create volumebackup

What did you expect to see?

  1. volumebackup success

What did you see instead?

  1. volume failed
  2. suspect that the pod was restarted for some reason, which caused the backup to be executed twice. The start time of the job is 8:41, the start time of the pod is 8:44, and the upload time of the backupmeta file is 8:42.

I0704 12:44:04.972263 9 backup.go:292] [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] I0704 12:44:04.972576 9 backup.go:292] [2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] I0704 12:44:04.977361 9 backup.go:292] I0704 12:44:04.977393 9 backup.go:299] Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument E0704 12:44:04.977497 9 manager.go:367] backup cluster backup-a/fed-backup-20230704-full-zm-dataplane-a data failed, err: cluster backup-a/fed-backup-20230704-full-zm-dataplane-a, wait pipe message failed, errMsg [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument , err: exit status 1 I0704 12:44:04.990386 9 backup_status_updater.go:123] Backup: [backup-a/fed-backup-20230704-full-zm-dataplane-a] updated successfully E0704 12:44:05.225639 9 backupsize.go:156] failed to get snapshots size 0, number of api request 0 W0704 12:44:05.225665 9 manager.go:324] Failed to calc volume snapshot backup size 0 bytes, ResourceNotFoundException: The snapshot 'snap-0a11162eaef40f5be' does not exist. { RespMetadata: { StatusCode: 404, RequestID: "56247a5c-9d56-4a50-ab97-6ffd8ccc0ddf" }, Message: "The snapshot 'snap-0a11162eaef40f5be' does not exist.", Reason: "SNAPSHOT_NOT_FOUND" } error: cluster backup-a/fed-backup-20230704-full-zm-dataplane-a, wait pipe message failed, errMsg [2023/07/04 12:44:04.972 +00:00] [ERROR] [backup.go:37] ["failed to backup"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:37\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [2023/07/04 12:44:04.972 +00:00] [ERROR] [main.go:60] ["br failed"] [error="backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument"] [errorVerbose="[BR:Common:ErrInvalidArgument]invalid argument\nbackup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!\ngithub.com/pingcap/tidb/br/pkg/backup.(Client).SetStorageAndCheckNotInUse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:235\ngithub.com/pingcap/tidb/br/pkg/task.RunBackupEBS\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/backup_ebs.go:124\nmain.runBackupCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:36\nmain.newFullBackupCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/backup.go:117\ngithub.com/spf13/cobra.(Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] Error: backup meta file exists in s3://wangle-ebs-test-us-west-2/backup-20230704-full-zm-dataplane-a//backupmeta, there may be some backup files in the path already, please specify a correct backup directory!: [BR:Common:ErrInvalidArgument]invalid argument , err: exit status 1 zhongmin@zhongmindeMacBook-P

ti-chi-bot commented 2 hours ago

This issue is restored for #5143, it originally created at 2023-07-05 03:31:09.