Velero backs up only metadata and not the data itself (pvc data)

[devopsInterviews]

Hi all, We are experiencing issues with backing up and restoring data on our vanilla Kubernetes cluster running on vSphere 7.3 to a remote MinIO S3 server using Velero, the vSphere plugin, and the AWS plugin. We are seeking your assistance to resolve this issue.

Environment Details: Kubernetes Version: Vanilla Kubernetes Cluster (1.26.1) vSphere Version: 7.3 Backup Solution: Velero velero version: 1.13.2 Plugins Used: velero/velero-plugin-for-aws 1.6.1 vsphereveleroplugin/velero-plugin-for-vsphere 1.4.2 velero/velero-plugin-for-csi 0.7.1

Object Storage: MinIO S3 on a remote server Current Setup: We have deployed Velero with the following pods in the velero namespace:

kubectl get pods -n velero

NAME READY STATUS RESTARTS AGE
backup-driver-88bd6cd86-ktlh7 1/1 Running 0 13m
datamgr-for-vsphere-plugin-49dc8 1/1 Running 0 12m
datamgr-for-vsphere-plugin-7flrm 1/1 Running 0 12m
datamgr-for-vsphere-plugin-9xswz 1/1 Running 0 12m
datamgr-for-vsphere-plugin-lv78l 1/1 Running 0 12m
velero-9c8d49fbc-6ndgp 1/1 Running 0 14m

Issue Details: Backup Process: We are able to initiate backups using Velero, and the metadata (e.g., namespaces, services, pods, PVC definitions) appears to be backed up correctly. Restore Process: During the restore process, all the Kubernetes objects (services, pods, PVCs, etc.) are restored successfully. However, the actual data inside the PVCs is not restored. The PVCs remain empty.

Observations:

Metadata Backup: The metadata is correctly migrated and visible after restoration. Data Absence: The data within the PVCs is not restored, resulting in empty PVCs even though the PVC objects themselves are recreated. Pod Logs: We have examined the logs of the datamgr-for-vsphere-plugin pods and observed errors related to invalid object names during the upload process to MinIO. Log Excerpt from datamgr-for-vsphere-plugin Pod:

time="2024-06-26T12:59:28Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=upload error="Failed to upload snapshot, ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc, to durable object storage. XMinioInvalidObjectName: Object name contains unsupported characters.\n\tstatus code: 400, request id: 17DC8F5886D5214C, host id: dd9025bab4ad464b049177c46595eb6ebf374d3b3fd1af9251148b658df7ac2e3e8" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:380" error.function="github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller.(*uploadController).processUpload" key=velero/upload-2d9093bb-3e07-4e19-ab94-ae97e614a2dc logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/generic_controller.go:138"

Log Excerpt from Restore Command:

error preparing persistentvolumeclaims/mynamespace/pvc-prometheus: rpc error: code = Unknown desc = Failed to create a CloneFromSnapshot CR: Phase=Failed, err=cloneFromSnapshot: Failed at calling SnapshotManager CreateVolumeFromSnapshotWithMetadata with peId pvc:mynamespace/pvc-prometheus, err: InternalError: We encountered an internal error, please try again.: cause(The filename, directory name, or volume label syntax is incorrect.)\n\tstatus code: 500, request id: 17DC8CD5B1B0338C, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8

Our Understanding of the Backup and Restore Process: Backup Process:

Velero initiates a backup process, capturing both metadata and data snapshots. The metadata is successfully backed up and stored in MinIO. Data snapshots (IVDs) are taken but fail to upload to MinIO due to invalid object names. Restore Process:

Velero restores the metadata correctly. Attempts to restore data from MinIO fail, leading to empty PVCs.

Installation Command: Here is the command we used to install Velero:

velero install --provider aws --bucket velero --secret-file credentials-velero --plugins velero/velero-plugin-for-aws:v1.6.1,vsphereveleroplugin/velero-plugin-for-vsphere:v1.4.2,velero/velero-plugin-for-csi:v0.7.1 --snapshot-location-config region=minio --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://:9000

We seek your guidance on the following points:

Resolution for Invalid Object Names: How can we ensure that object names generated during the snapshot process are valid and compatible with MinIO? We don't have control over them as far as I know. Ensuring Data Backup and Restore: Steps we can take to ensure that both metadata and data are correctly backed up and restored. We appreciate any insights or recommendations you can provide to help us resolve this issue.

Thank you for your support.

Best regards !

[blackpiglet]

@xing-yang Could you please help to take a look at this issue?

[devopsInterviews]

@blackpiglet @xing-yang We would really appreciate any help.

[blackpiglet]

The vSphere plugin issue needs expertise from @xing-yang.

I have one suggestion too. Please remove the CSI plugin from the environment, although this may not be the cause of your issue.