search

pingcap / br

A command-line tool for distributed backup and restoration of the TiDB cluster data
https://pingcap.com/docs/dev/how-to/maintain/backup-and-restore/br/
Apache License 2.0
123 stars 102 forks source link

br_log_restore fails with `start ts:421355834788020227 is greater than resolved ts:0` #641

Open overvenus opened 3 years ago

overvenus commented 3 years ago

br_log_restore fails https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/br_ghpr_unit_and_integration_test/detail/br_ghpr_unit_and_integration_test/4116/pipeline#step-1318-log-326

``` [2020-12-07T12:43:29.384Z] [2020/12/07 20:43:26.743 +08:00] [ERROR] [restore.go:37] ["failed to restore"] [error="start ts:421355834788020227 is greater than resolved ts:0: [BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation"] [errorVerbose="[BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation\nstart ts:421355834788020227 is greater than resolved ts:0\ngithub.com/pingcap/br/pkg/restore.(*LogClient).RestoreLogData\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/restore/log_client.go:938\ngithub.com/pingcap/br/pkg/task.RunLogRestore\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/task/restore_log.go:136\ngithub.com/pingcap/br/cmd.runLogRestoreCommand\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:36\ngithub.com/pingcap/br/cmd.newLogRestoreCommand.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:132\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\ngithub.com/pingcap/br.main\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:58\ngithub.com/pingcap/br.TestRunMain.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/br/cmd.runLogRestoreCommand\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:37\ngithub.com/pingcap/br/cmd.newLogRestoreCommand.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:132\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\ngithub.com/pingcap/br.main\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:58\ngithub.com/pingcap/br.TestRunMain.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39"] [2020-12-07T12:43:29.384Z] Error: start ts:421355834788020227 is greater than resolved ts:0: [BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation [2020-12-07T12:43:29.384Z] [2020/12/07 20:43:26.743 +08:00] [ERROR] [main.go:59] ["br failed"] [error="start ts:421355834788020227 is greater than resolved ts:0: [BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation"] [errorVerbose="[BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation\nstart ts:421355834788020227 is greater than resolved ts:0\ngithub.com/pingcap/br/pkg/restore.(*LogClient).RestoreLogData\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/restore/log_client.go:938\ngithub.com/pingcap/br/pkg/task.RunLogRestore\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/task/restore_log.go:136\ngithub.com/pingcap/br/cmd.runLogRestoreCommand\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:36\ngithub.com/pingcap/br/cmd.newLogRestoreCommand.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:132\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\ngithub.com/pingcap/br.main\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:58\ngithub.com/pingcap/br.TestRunMain.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/br.main\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:59\ngithub.com/pingcap/br.TestRunMain.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39"] ```
3pointer commented 3 years ago

https://github.com/pingcap/br/pull/636#issuecomment-740344957 This error usually means cdc didn't flush meta to minio. it seems we need sleep more time here...