Open raghulkrishna opened 3 years ago
Hi @raghulkrishna - I suspect this issue is related to https://github.com/vmware-tanzu/velero/issues/4108, however, given that you are using restic I'm not sure what the behavior should be. Could you please provide the following?
kubectl logs deployment/velero -n velero
velero backup describe <backupname>
or kubectl get backup/<backupname> -n velero -o yaml
velero backup logs <backupname>
velero restore describe <restorename> --details
velero restore logs <restorename>
Please also provide the pod logs from each of the restic pods. Thanks!
velero backup describe
Phase: Completed
Errors: 0 Warnings: 1
Namespaces:
Included: restic-test
Excluded:
Resources:
Included: *
Excluded:
Label selector:
Storage Location: restic
Velero-Native Snapshot PVs: auto
TTL: 720h0m0s
Hooks:
Backup Format Version: 1.1.0
Started: 2021-09-21 09:55:07 +0200 CEST Completed: 2021-09-21 09:55:51 +0200 CEST
Expiration: 2021-10-21 09:55:07 +0200 CEST
Total items to be backed up: 49 Items backed up: 49
Velero-Native Snapshots:
Restic Backups (specify --details for more information): Completed: 1
velero backup logs
time="2021-09-21T07:55:07Z" level=info msg="Setting up backup temp file" backup=velero/efs-csi-pvc-test-2 logSource="pkg/controller/backup_controller.go:556"
time="2021-09-21T07:55:07Z" level=info msg="Setting up plugin manager" backup=velero/efs-csi-pvc-test-2 logSource="pkg/controller/backup_controller.go:563"
time="2021-09-21T07:55:07Z" level=info msg="Getting backup item actions" backup=velero/efs-csi-pvc-test-2 logSource="pkg/controller/backup_controller.go:567"
time="2021-09-21T07:55:07Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/efs-csi-pvc-test-2 logSource="pkg/controller/backup_controller.go:573"
time="2021-09-21T07:55:08Z" level=info msg="Writing backup version file" backup=velero/efs-csi-pvc-test-2 logSource="pkg/backup/backup.go:212"
time="2021-09-21T07:55:08Z" level=info msg="Including namespaces: restic-test" backup=velero/efs-csi-pvc-test-2 logSource="pkg/backup/backup.go:218"
time="2021-09-21T07:55:08Z" level=info msg="Excluding namespaces:
velero restore describe
Name: efs-csi-pvc-test-1
Namespace: velero
Labels:
Phase: PartiallyFailed (run 'velero restore logs efs-csi-pvc-test-1' for more information) Total items to be restored: 6 Items restored: 6
Started: 2021-09-21 09:38:19 +0200 CEST Completed: 2021-09-21 09:38:28 +0200 CEST
Errors: Velero: pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=s3:s3-eu-central-1.amazonaws.com/eks-staging-backups-restic/restic/restic-test --password-file=/tmp/credentials/velero/velero-restic-credentials-repository-password --cache-dir=/scratch/.cache/restic b9e269be --target=., stdout=restoring <Snapshot b9e269be of [/host_pods/d13e1044-bafa-47c5-9b4b-4e5d20ff9955/volumes/kubernetes.io~csi/pvc-b8f98e77-c880-450c-8d65-d138003d1364/mount] at 2021-09-21 07:34:09.812061606 +0000 UTC by root@velero> to . , stderr=ignoring error for /out: Lchown: lchown /host_pods/341eb214-b448-4e95-a25f-d04d911ead3d/volumes/kubernetes.io~csi/pvc-70eccfd6-c365-4b0d-85d2-911b0aa48f11/mount/out: operation not permitted Fatal: There were 1 errors
: exit status 1
Cluster:
Backup: efs-csi-pvc-test-1
Namespaces:
Included: all namespaces found in the backup
Excluded:
Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io Cluster-scoped: auto
Namespace mappings:
Label selector:
Restore PVs: auto
Restic Restores: Failed: restic-test/efs-app: persistent-storage
Preserve Service NodePorts: auto logs: time="2021-09-21T07:38:19Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:465" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Starting restore of backup velero/efs-csi-pvc-test-1" logSource="pkg/restore/restore.go:387" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:1674" resource=volumesnapshotclass.snapshot.storage.k8s.io restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:1674" resource=volumesnapshotcontents.snapshot.storage.k8s.io restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:1674" resource=volumesnapshots.snapshot.storage.k8s.io restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Resource 'persistentvolumes' will be restored at cluster scope" logSource="pkg/restore/restore.go:1763" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Resource 'persistentvolumeclaims' will be restored into namespace 'restic-test'" logSource="pkg/restore/restore.go:1761" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Resource 'secrets' will be restored into namespace 'restic-test'" logSource="pkg/restore/restore.go:1761" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Resource 'configmaps' will be restored into namespace 'restic-test'" logSource="pkg/restore/restore.go:1761" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Resource 'serviceaccounts' will be restored into namespace 'restic-test'" logSource="pkg/restore/restore.go:1761" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Resource 'pods' will be restored into namespace 'restic-test'" logSource="pkg/restore/restore.go:1761" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:1674" resource=clusters.cluster.x-k8s.io restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:1674" resource=clusterresourcesets.addons.cluster.x-k8s.io restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:1691" resource=events restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:883" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Dynamically re-provisioning persistent volume because it has a restic backup to be restored." logSource="pkg/restore/restore.go:1092" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restored 1 items out of an estimated total of 6 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=pvc-b8f98e77-c880-450c-8d65-d138003d1364 namespace= progress= resource=persistentvolumes restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Getting client for /v1, Kind=PersistentVolumeClaim" logSource="pkg/restore/restore.go:883" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1133" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Adding PV pvc-b8f98e77-c880-450c-8d65-d138003d1364 as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Skipping persistentvolumes/pvc-b8f98e77-c880-450c-8d65-d138003d1364 because it's already been restored." logSource="pkg/restore/restore.go:991" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1133" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing ChangePVCNodeSelectorAction" cmd=/velero logSource="pkg/restore/change_pvc_node_selector.go:65" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Done executing ChangePVCNodeSelectorAction" cmd=/velero logSource="pkg/restore/change_pvc_node_selector.go:93" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1133" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:65" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:76" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Resetting PersistentVolumeClaim restic-test/efs-claim for dynamic provisioning" logSource="pkg/restore/restore.go:1212" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Attempting to restore PersistentVolumeClaim: efs-claim" logSource="pkg/restore/restore.go:1238" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restored 2 items out of an estimated total of 6 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=efs-claim namespace=restic-test progress= resource=persistentvolumeclaims restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Getting client for /v1, Kind=Secret" logSource="pkg/restore/restore.go:883" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Attempting to restore Secret: default-token-lsmbx" logSource="pkg/restore/restore.go:1238" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restore of Secret, default-token-lsmbx skipped: it already exists in the cluster and is the same as the backed up version" logSource="pkg/restore/restore.go:1296" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restored 3 items out of an estimated total of 6 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=default-token-lsmbx namespace=restic-test progress= resource=secrets restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Getting client for /v1, Kind=ConfigMap" logSource="pkg/restore/restore.go:883" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Attempting to restore ConfigMap: kube-root-ca.crt" logSource="pkg/restore/restore.go:1238" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restore of ConfigMap, kube-root-ca.crt skipped: it already exists in the cluster and is the same as the backed up version" logSource="pkg/restore/restore.go:1296" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restored 4 items out of an estimated total of 6 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=kube-root-ca.crt namespace=restic-test progress= resource=configmaps restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Getting client for /v1, Kind=ServiceAccount" logSource="pkg/restore/restore.go:883" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing item action for serviceaccounts" logSource="pkg/restore/restore.go:1133" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing ServiceAccountAction" cmd=/velero logSource="pkg/restore/service_account_action.go:47" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Done executing ServiceAccountAction" cmd=/velero logSource="pkg/restore/service_account_action.go:78" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Attempting to restore ServiceAccount: default" logSource="pkg/restore/restore.go:1238" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restored 5 items out of an estimated total of 6 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=default namespace=restic-test progress= resource=serviceaccounts restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Getting client for /v1, Kind=Pod" logSource="pkg/restore/restore.go:883" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1133" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing AddPVCFromPodAction" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:44" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Adding PVC restic-test/efs-claim as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pvc_from_pod_action.go:58" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Skipping persistentvolumeclaims/restic-test/efs-claim because it's already been restored." logSource="pkg/restore/restore.go:991" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1133" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing InitRestoreHookPodAction" cmd=/velero logSource="pkg/restore/init_restorehook_pod_action.go:49" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Pod restic-test/efs-app has no init.hook.restore.velero.io/container-image annotation, no initRestoreHook in annotation" cmd=/velero logSource="internal/hook/item_hook_handler.go:350" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Handling InitRestoreHooks from RestoreSpec" cmd=/velero logSource="internal/hook/item_hook_handler.go:138" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Returning pod restic-test/efs-app with 0 init container(s)" cmd=/velero logSource="internal/hook/item_hook_handler.go:157" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Returning from InitRestoreHookPodAction" cmd=/velero logSource="pkg/restore/init_restorehook_pod_action.go:57" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1133" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1133" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:71" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restic backups for pod found" cmd=/velero logSource="pkg/restore/restic_restore_action.go:106" pluginName=velero pod=restic-test/efs-app restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Using image \"velero/velero-restic-restore-helper:v1.6.1\"" cmd=/velero logSource="pkg/restore/restic_restore_action.go:117" pluginName=velero pod=restic-test/efs-app restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Done executing ResticRestoreAction" cmd=/velero logSource="pkg/restore/restic_restore_action.go:175" pluginName=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Attempting to restore Pod: efs-app" logSource="pkg/restore/restore.go:1238" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Restored 6 items out of an estimated total of 6 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=efs-app namespace=restic-test progress= resource=pods restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:20Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:546" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:28Z" level=error msg="unable to successfully complete restic restores of pod's volumes" error="pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=s3:s3-eu-central-1.amazonaws.com/eks-staging-backups-restic/restic/restic-test --password-file=/tmp/credentials/velero/velero-restic-credentials-repository-password --cache-dir=/scratch/.cache/restic b9e269be --target=., stdout=restoring <Snapshot b9e269be of [/host_pods/d13e1044-bafa-47c5-9b4b-4e5d20ff9955/volumes/kubernetes.io~csi/pvc-b8f98e77-c880-450c-8d65-d138003d1364/mount] at 2021-09-21 07:34:09.812061606 +0000 UTC by root@velero> to .\n, stderr=ignoring error for /out: Lchown: lchown /host_pods/341eb214-b448-4e95-a25f-d04d911ead3d/volumes/kubernetes.io~csi/pvc-70eccfd6-c365-4b0d-85d2-911b0aa48f11/mount/out: operation not permitted\nFatal: There were 1 errors\n\n: exit status 1" logSource="pkg/restore/restore.go:1438" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:28Z" level=info msg="Done waiting for all restic restores to complete" logSource="pkg/restore/restore.go:562" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:28Z" level=info msg="Waiting for all post-restore-exec hooks to complete" logSource="pkg/restore/restore.go:566" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:28Z" level=info msg="Done waiting for all post-restore exec hooks to complete" logSource="pkg/restore/restore.go:574" restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:28Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:480" restore=velero/efs-csi-pvc-test-1
logs from restic pod
time="2021-09-21T07:34:09Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/efs-csi-pvc-test-1 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:277" name=efs-csi-pvc-test-1-c25nd namespace=velero time="2021-09-21T07:34:12Z" level=info msg="Backup completed" backup=velero/efs-csi-pvc-test-1 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:330" name=efs-csi-pvc-test-1-c25nd namespace=velero time="2021-09-21T07:38:26Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:280" name=efs-csi-pvc-test-1-mfhps namespace=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:38:28Z" level=error msg="Error restoring volume" controller=pod-volume-restore error="error running restic restore, cmd=restic restore --repo=s3:s3-eu-central-1.amazonaws.com/eks-staging-backups-restic/restic/restic-test --password-file=/tmp/credentials/velero/velero-restic-credentials-repository-password --cache-dir=/scratch/.cache/restic b9e269be --target=., stdout=restoring <Snapshot b9e269be of [/host_pods/d13e1044-bafa-47c5-9b4b-4e5d20ff9955/volumes/kubernetes.io~csi/pvc-b8f98e77-c880-450c-8d65-d138003d1364/mount] at 2021-09-21 07:34:09.812061606 +0000 UTC by root@velero> to .\n, stderr=ignoring error for /out: Lchown: lchown /host_pods/341eb214-b448-4e95-a25f-d04d911ead3d/volumes/kubernetes.io~csi/pvc-70eccfd6-c365-4b0d-85d2-911b0aa48f11/mount/out: operation not permitted\nFatal: There were 1 errors\n\n: exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/pod_volume_restore_controller.go:378" error.function="github.com/vmware-tanzu/velero/pkg/controller.(podVolumeRestoreController).restorePodVolume" logSource="pkg/controller/pod_volume_restore_controller.go:308" name=efs-csi-pvc-test-1-mfhps namespace=velero restore=velero/efs-csi-pvc-test-1 time="2021-09-21T07:55:48Z" level=info msg="Backup starting" backup=velero/efs-csi-pvc-test-2 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=efs-csi-pvc-test-2-8fjmk namespace=velero time="2021-09-21T07:55:48Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/efs-csi-pvc-test-2 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:340" name=efs-csi-pvc-test-2-8fjmk namespace=velero pvcUID=947081e4-3984-4688-8ebf-6b21fdca56df time="2021-09-21T07:55:48Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/efs-csi-pvc-test-2 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:370" name=efs-csi-pvc-test-2-8fjmk namespace=velero pvcUID=947081e4-3984-4688-8ebf-6b21fdca56df time="2021-09-21T07:55:48Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/efs-csi-pvc-test-2 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:277" name=efs-csi-pvc-test-2-8fjmk namespace=velero time="2021-09-21T07:55:51Z" level=info msg="Backup completed" backup=velero/efs-csi-pvc-test-2 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:330" name=efs-csi-pvc-test-2-8fjmk namespace=velero time="2021-09-21T08:02:33Z" level=info msg="Restore starting" controller=pod-volume-restore logSource="pkg/controller/pod_volume_restore_controller.go:280" name=efs-csi-pvc-test-2-c7jjf namespace=velero restore=velero/efs-csi-pvc-test-2 time="2021-09-21T08:02:36Z" level=error msg="Error restoring volume" controller=pod-volume-restore error="error running restic restore, cmd=restic restore --repo=s3:s3-eu-central-1.amazonaws.com/eks-staging-backups-restic/restic/restic-test --password-file=/tmp/credentials/velero/velero-restic-credentials-repository-password --cache-dir=/scratch/.cache/restic 8a5dea07 --target=., stdout=restoring <Snapshot 8a5dea07 of [/host_pods/e1a11c4a-b70b-4102-8ea3-f9b9e9428bd5/volumes/kubernetes.io~csi/pvc-947081e4-3984-4688-8ebf-6b21fdca56df/mount] at 2021-09-21 07:55:48.855551769 +0000 UTC by root@velero> to .\n, stderr=ignoring error for /o.zip: Lchown: lchown /host_pods/2ef2f0d9-0914-4c69-91ca-8170955c5f4e/volumes/kubernetes.io~csi/pvc-fc8bc62d-fa7b-479b-a118-68ebc36b46d8/mount/o.zip: operation not permitted\nignoring error for /out: Lchown: lchown /host_pods/2ef2f0d9-0914-4c69-91ca-8170955c5f4e/volumes/kubernetes.io~csi/pvc-fc8bc62d-fa7b-479b-a118-68ebc36b46d8/mount/out: operation not permitted\nFatal: There were 2 errors\n\n: exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/pod_volume_restore_controller.go:378" error.function="github.com/vmware-tanzu/velero/pkg/controller.(podVolumeRestoreController).restorePodVolume" logSource="pkg/controller/pod_volume_restore_controller.go:308" name=efs-csi-pvc-test-2-c7jjf namespace=velero restore=velero/efs-csi-pvc-test-2 Logs from 9/19/2021, 9:48:44 PM
@raghulkrishna please use a gist instead of pasting the full log as comments of the issue, currently this is unreadable.
@raghulkrishna
It seems like a permission problem on the efs mount and could you check if the user restic pod uses has permission to chown
it?
@reasonerjt can you please suggest me how can I check that?
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: efs-sc provisioner: efs.csi.aws.com parameters: provisioningMode: efs-ap fileSystemId: fs-XXXXXX directoryPerms: "700"
This is how i created the storage class
What steps did you take and what happened:
Storage class dynamic provisioning aws efs
created backup using
output
created restore