reasonerjt
commented
2 years ago Checking the log of the back, there are many errors like this:
time="2021-08-26T16:30:34Z" level=error msg="Error backing up item" backup=management/backup-20210826092545 error="pod volume backup failed: error running restic backup, stderr=
{\"message_type\":\"error\",\"error\":{\"Op\":\"lstat\",\"Path\":\"nodes/0/indices/Xb1jihNcRR-wML2VEubkMA/0/index/_284.fdm\",\"Err\":2},\"during\":\"archival\",\"item\":\"/host_pods/147ad440-f30c-4102-840f-375bf5410f24/volumes/kubernetes.io~csi/pvc-771113e7-6153-4fad-83df-dd4adae5bc89/mount/nodes/0/indices/Xb1jihNcRR-wML2VEubkMA/0/index/_284.fdm\"}
\n
{\"message_type\":\"error\",\"error\":{\"Op\":\"lstat\",\"Path\":\"nodes/0/indices/Xb1jihNcRR-wML2VEubkMA/0/index/_284.fdt\",\"Err\":2},\"during\":\"archival\",\"item\":\"/host_pods/147ad440-f30c-4102-840f-375bf5410f24/volumes/kubernetes.io~csi/pvc-771113e7-6153-4fad-83df-dd4adae5bc89/mount/nodes/0/indices/Xb1jihNcRR-wML2VEubkMA/0/index/_284.fdt\"}\
......So the problem is that the restic backup failed.
Could you check the log of restic daemon set at the same time frame?
looks very similar to #4065
I'm currently not quite sure how restic works but I notice during:archival
I'll double check if it mean restic may scan the files first then try to archive the files, and during that timeframe the files are gone.
nyarly
dsu-igeek
crdemon09
What steps did you take and what happened:
I created a cluster to restore a backup into. Ran a restore; the Velero restore worked fine, but no Restic volumes were restored. Tore down restore cluster and replaced it, without restoring.
Based on https://github.com/vmware-tanzu/velero/blob/main/site/content/docs/v1.5/restic.md#troubleshooting: Neither instance of the restore cluster has ResticRepository objects, despite the source cluster having repositories, and all 3 clusters being deployed in the same GCP project. The restore clusters have PodVolumeBackup resources that match the source cluster.
What did you expect to happen:
Volumes backed up with Restic to be restored.
The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other pastebin is fine.) Velero is deployed into a different namespace in these clusters.
kubectl logs deployment/velero -n managementhttps://gist.github.com/67a8e2d002b6426b1b531d38e31328fbvelero -n management backup describe <backupname>https://gist.github.com/5d3d6892fd7ee5f4f432f9765b9c9358velero -n management backup logs <backupname>https://gist.github.com/2a3c4e28360f2e761a06edd7c03ef793This instance of the restore cluster hasn't had a restore; I'll update after re-attempting.
velero -n management restore describe <restorename>velero -n management restore logs <restorename>Environment:
velero version):velero client config get features):features: <NOT SETkubectl version):/etc/os-release): GCP UbuntuVote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.