shubham-pampattiwar
commented
2 months ago Can you please double check if you have the latest velero CRDs ? (specially restore CRD in this case) We have seen this error in cases where the CRDs did not match the velero server version.
justsomecorporateuser
What steps did you take and what happened: creating a regression test for our Velero installation. Test creates a backup, then cleans few deployments and finally restores from backup.
Restore ends up in being in progress, forever. Logs show that all items to be restored were handled:
time="2024-08-06T14:21:33Z" level=info msg="Restored 170 items out of an estimated total of 170 (estimate will change throughout the restore) time="2024-08-06T14:21:33Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:605" restore=velero/velero-test-restore-20240806142022
Right after that log shows error (the last line):
time="2024-08-06T14:21:33Z" level=debug msg="looking for plugin in registry" kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:133" name=velero.io/gcp restore=velero/velero-test-restore-20240806142022
time="2024-08-06T14:21:33Z" level=debug msg="found preexisting restartable plugin process" command=/plugins/velero-plugin-for-gcp kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:144" name=velero.io/gcp restore=velero/velero-test-restore-20240806142022
time="2024-08-06T14:21:34Z" level=debug msg="Restore Finalizing" logSource="pkg/controller/restore_controller.go:672"
time="2024-08-06T14:21:34Z" level=debug msg="received EOF, stopping recv loop" cmd=/plugins/velero-plugin-for-gcp err="rpc error: code = Unavailable desc = error reading from server: EOF" logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" pluginName=stdio restore=velero/velero-test-restore-20240806142022
time="2024-08-06T14:21:34Z" level=info msg="plugin process exited" cmd=/plugins/velero-plugin-for-gcp id=383 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-gcp restore=velero/velero-test-restore-20240806142022
time="2024-08-06T14:21:34Z" level=debug msg="plugin exited" cmd=/plugins/velero-plugin-for-gcp logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" restore=velero/velero-test-restore-20240806142022
time="2024-08-06T14:21:34Z" level=debug msg="received EOF, stopping recv loop" cmd=/velero err="rpc error: code = Unavailable desc = error reading from server: EOF" logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" pluginName=stdio restore=velero/velero-test-restore-20240806142022
INFO 2024-08-06T14:21:34.687417674Z [resource.labels.containerName: velero] time="2024-08-06T14:21:34Z" level=info msg="plugin process exited" cmd=/velero id=395 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/velero restore=velero/velero-test-restore-20240806142022
time="2024-08-06T14:21:34Z" level=debug msg="plugin exited" cmd=/velero logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" restore=velero/velero-test-restore-20240806142022
time="2024-08-06T14:21:34Z" level=debug msg="Updating restore's final status" Restore=velero/velero-test-restore-20240806142022 logSource="pkg/controller/restore_controller.go:277"
time="2024-08-06T14:21:34Z" level=info msg="Error updating restore's final status" Restore=velero/velero-test-restore-20240806142022 error="Restore.velero.io \"velero-test-restore-20240806142022\" is invalid: status.phase: Unsupported value: \"Finalizing\": supported values: \"New\", \"FailedValidation\", \"InProgress\", \"WaitingForPluginOperations\", \"WaitingForPluginOperationsPartiallyFailed\", \"Completed\", \"PartiallyFailed\", \"Failed\"" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:280" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*restoreReconciler).Reconcile" logSource="pkg/controller/restore_controller.go:280"
Environment: