Issue with Data Backup and Restore on vSphere using Velero vSphere Plugin

[devopsInterviews]

Dear Velero vSphere Plugin Team,

We are experiencing issues with backing up and restoring data on our vanilla Kubernetes cluster running on vSphere 7.3 to a remote MinIO S3 server using Velero, the vSphere plugin, and the AWS plugin. We are seeking your assistance to resolve this issue.

Environment Details: Kubernetes Version: Vanilla Kubernetes Cluster (1.26.1) vSphere Version: 7.3 velero version: 1.13.2 Plugins Used: velero/velero-plugin-for-aws 1.6.1 vsphereveleroplugin/velero-plugin-for-vsphere 1.4.2 velero/velero-plugin-for-csi 0.7.1

Object Storage: MinIO S3 on a remote server Current Setup: We have deployed Velero with the following pods in the velero namespace:

kubectl get pods -n velero

NAME READY STATUS RESTARTS AGE backup-driver-88bd6cd86-ktlh7 1/1 Running 0 13m datamgr-for-vsphere-plugin-49dc8 1/1 Running 0 12m datamgr-for-vsphere-plugin-7flrm 1/1 Running 0 12m datamgr-for-vsphere-plugin-9xswz 1/1 Running 0 12m datamgr-for-vsphere-plugin-lv78l 1/1 Running 0 12m velero-9c8d49fbc-6ndgp 1/1 Running 0 14m

Issue Details: Backup Process: We are able to initiate backups using Velero, and the metadata (e.g., namespaces, services, pods, PVC definitions) appears to be backed up correctly. Restore Process: During the restore process, all the Kubernetes objects (services, pods, PVCs, etc.) are restored successfully. However, the actual data inside the PVCs is not restored. The PVCs remain empty.

Observations:

Metadata Backup: The metadata is correctly migrated and visible after restoration. Data Absence: The data within the PVCs is not restored, resulting in empty PVCs even though the PVC objects themselves are recreated. Pod Logs: We have examined the logs of the datamgr-for-vsphere-plugin pods and observed errors related to invalid object names during the upload process to MinIO. Log Excerpt from datamgr-for-vsphere-plugin Pod:

time="2024-06-26T12:59:28Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=upload error="Failed to upload snapshot, ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc, to durable object storage. XMinioInvalidObjectName: Object name contains unsupported characters.\n\tstatus code: 400, request id: 17DC8F5886D5214C, host id: dd9025bab4ad464b049177c46595eb6ebf374d3b3fd1af9251148b658df7ac2e3e8" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:380" error.function="github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller.(*uploadController).processUpload" key=velero/upload-2d9093bb-3e07-4e19-ab94-ae97e614a2dc logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/generic_controller.go:138"

Log Excerpt from Restore Command:

error preparing persistentvolumeclaims/mynamespace/pvc-prometheus: rpc error: code = Unknown desc = Failed to create a CloneFromSnapshot CR: Phase=Failed, err=cloneFromSnapshot: Failed at calling SnapshotManager CreateVolumeFromSnapshotWithMetadata with peId pvc:mynamespace/pvc-prometheus, err: InternalError: We encountered an internal error, please try again.: cause(The filename, directory name, or volume label syntax is incorrect.)\n\tstatus code: 500, request id: 17DC8CD5B1B0338C, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8

Our Understanding of the Backup and Restore Process: Backup Process:

Velero initiates a backup process, capturing both metadata and data snapshots. The metadata is successfully backed up and stored in MinIO. Data snapshots (IVDs) are taken but fail to upload to MinIO due to invalid object names. Restore Process:

Velero restores the metadata correctly. Attempts to restore data from MinIO fail, leading to empty PVCs.

Installation Command: Here is the command we used to install Velero:

velero install --provider aws --bucket velero --secret-file credentials-velero --plugins velero/velero-plugin-for-aws:v1.6.1,vsphereveleroplugin/velero-plugin-for-vsphere:v1.4.2,velero/velero-plugin-for-csi:v0.7.1 --snapshot-location-config region=minio --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://:9000

We seek your guidance on the following points:

Resolution for Invalid Object Names: How can we ensure that object names generated during the snapshot process are valid and compatible with MinIO? We don't have control over them as far as I know. Ensuring Data Backup and Restore: Steps we can take to ensure that both metadata and data are correctly backed up and restored. We appreciate any insights or recommendations you can provide to help us resolve this issue.

Thank you for your support.

Best regards !

[devopsInterviews]

Anyone?

[BarelElbaz]

https://github.com/vmware-tanzu/velero/issues/7938#issuecomment-2198574300

[BarelElbaz]

Just for any case I put here the details of the bug: We have further investigation: There is a difference between running it on Windows and Linux hosts, it seems that the issue occurs only on Windows (Running Minio as exe) To reproduce, we used the latest versions of the Velero plugin and Minio, and here are the results: On linux:

 pip3 install minio
 echo hello > file.txt
 #Create a "velero" bucket in minio

Run the following script

from minio import Minio
from minio.error import S3Error

def main():
    # Initialize minio client object.
    minioClient = Minio(
        "localhost:9000",
        access_key="minioadmin",
        secret_key="minioadmin",
        secure=False
    )

    # Define the object name with unsupported characters.
    object_name = "ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc"
    bucket_name = "velero"
    file_path = "file.txt"
    content_type = "application/text"

    # Upload the file
    try:
        minioClient.fput_object(bucket_name, object_name, file_path, content_type=content_type)
        print(f"Successfully uploaded {object_name} to bucket {bucket_name}")
    except S3Error as err:
        print(f"Error occurred: {err}")

if __name__ == "__main__":
    main()

The upload passed successfully:

 Successfully uploaded ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc to bucket velero

Try the same on a Windows host: Install minio as exe (https://dl.min.io/server/minio/release/windows-amd64/minio.exe) and run the script:

C:\minio>C:\minio\myscript.py
Error occurred: S3 operation failed; code: XMinioInvalidObjectName, message: Object name contains unsupported characters., resource: /velero/ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc, request_id: 17DDCC6B9550F1A0, host_id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, bucket_name: velero, object_name: ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc

It's probably an issue with minio's windows flavor because of a limitation in the NTFS file system (which doesn't support colon ':' in filenames):

Here is the code that verifies it in Minio: https://github.com/minio/minio/blob/f736702da8f3fe0967f9a25bab6c8d7e27d002a4/cmd/object-api-utils.go#L197-L223

[Tomer13]

The same problem is occurring to me as well. I have a Minio service on a Windows server that functions as a backup server. I encountered the same issue with backing up process with Velero and Vsphere plugin . Any update from your side to handle this issue and support Windows machine in the vSphere plugin?

[xing-yang]

Sorry, we don't have support for Windows yet. I'll add this to our backlog.