blackpiglet
commented
1 month ago There are some warning and error information found from Velero pod log. IMO, this is still related to the block device backend. There should be some information printed in the Linstor's snapshot controller.
time="2024-07-15T20:39:55Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to take snapshot of the volume pvc-e384a2d9-5a9c-40fb-be46-feed10269ce3: \"rpc error: code = Internal desc = tried deleting a leftover unsuccessful snapshot\". Snapshot controller will retry later." cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshot_action.go:318" pluginName=velero
time="2024-07-15T20:40:35Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to take snapshot of the volume pvc-e384a2d9-5a9c-40fb-be46-feed10269ce3: \"rpc error: code = Internal desc = failed to create snapshot: error creating S3 backup: Message: '(Node: 'ovbh-vtest-k8s02-worker01') Shutdown of the DRBD resource 'pvc-e384a2d9-5a9c-40fb-be46-feed10269ce3 failed'; Cause: 'The external command for stopping the DRBD resource failed'; Correction: '- Check whether the required software is installed\\n- Check whether the application's search path includes the location\\n of the external software\\n- Check whether the application has execute permission for the external command\\n'; Reports: '[66958733-EF22C-000006]'\". Snapshot controller will retry later." cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshot_action.go:318" pluginName=velerotime="2024-07-15T20:42:47Z" level=error msg="VolumeSnapshot mongodb3/velero-mongod-data-preprod-mongodb-rs0-2-c8jkz is not ready. This is not expected." backup=velero/mongodb3-test1 cmd=/velero controller=backup-finalizer logSource="pkg/util/csi/volume_snapshot.go:534" pluginName=velero
time="2024-07-15T20:37:37Z" level=error msg="VolumeSnapshot api-cache3/velero-redis-data-preprod-api-cache-redis-node-0-tkfd4 is not ready. This is not expected." backup=velero/api-cache-test1 cmd=/velero controller=backup-finalizer logSource="pkg/util/csi/volume_snapshot.go:534" pluginName=velero
phoenix-bjoern
jonathon2nd
What steps did you take and what happened:
Some snapshots are not finishing for some reason. Some snapshots say completed, but missing from s3. For example,
pvc-99b78545-d39b-4502-be80-50eb3c4e428bis not present in s3. Issue does not seem to be k8s host or external linstor host related, there is not a correlation to pv/ k8s-worker errorsWhat did you expect to happen: All snapshots be present in s3
The following information will help us better understand what's going on: REPO URL: https://vmware-tanzu.github.io/helm-charts CHART: velero:7.1.0
If you are using velero v1.7.0+:
Please use
velero debug --backup <backupname> --restore <restorename>to generate the support bundle, and attach to this issue, more options please refer tovelero debug --helpThis is not working.postgres-backup-logs.txt mongodb-backup-logs.txt postgres-backup.txt mongodb-backup.txt preprod-velero-c57d98d86-65lhx_velero.log
Anything else you would like to add:
Only uploading for mongodb and postgres. Not including anything for tasks or api-cache that you may see in images.
Environment:
velero version):velero client config get features):features: EnableCSIkubectl version): v1.26.13 +rke2r1/etc/os-release): Rocky Linux 8.9 (Green Obsidian)Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.