search

vmware-tanzu / velero-plugin-for-vsphere

Plugin to support Velero on vSphere
Other
59 stars 49 forks source link

Restore failes for PVCs that have an ownerReference #579

Open Sediket opened 5 months ago

Sediket commented 5 months ago

Describe the bug Re-creating: https://github.com/vmware-tanzu/velero/issues/7862

I'm implementing velero as our backup solution and testing backups. I backed up a helm install of grafana's loki (https://github.com/grafana/loki/tree/main/production/helm/loki) and I noticed the PVCs have an ownerReference set to the loki-backend StatefulSet. So on restore the PVC will not be created referencing the ownerReference, same for all the PVCs in the StatefulSet. There are PVCs that don't have an ownerReference and they restore correctly.

As I understand Velero should be removing the metaData to include the ownerReference but it doesn't seem to be happening for some reason.

I also tested this by re-creating my storage class so it doesn't have to re-map the storage class on restore, but I still get the same result. The PVCs are empty and error, referencing the ownerReference is not present in the namespace.

Details in the attached issue.

swiftlee commented 5 months ago

Hello! I am working with @Sediket on this issue. I'm fairly new to Velero and its plugins, so I may be incorrect.

It looks like the the only metadata that is removed from PVCs at restore is health annotations as shown here: https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/main/pkg/plugin/restore_pvc_action_plugin.go#L97

It appears that all other metadata removal is omitted for PVCs specifically, and it does seem to override the parent behavior from Velero.

Given this, PVCs are keeping the ownerReference to their corresponding StatefulSet. However, when the PVCs attempt to restore they fail since no such owner exists in the mapped namespace.

xing-yang commented 5 months ago

Can you provide kubectl describe of the StatefulSet and PVCs that failed to be restored and the ones can be restored?

Sediket commented 5 months ago

Yes, here they are successfulPVC.log successfulSTS.log failedPVC.log failedSTS.log

xing-yang commented 5 months ago

Alright. Looks like this is a bug.