Migration between clusters not working as expected: snapshot not bound

What steps did you take and what happened:

Test migrating a simple workload (wordpress) from cluster A to cluster B, following instructions there : https://velero.io/docs/main/migration-case/ and there https://velero.io/docs/main/restore-reference/#changing-pvpvc-storage-classes
Cluster A workload consists of 2 openebs-zfs-localpv volumes (mariadb+wordpress) running on RKE2 1.30.1
Cluster B has linstor and openebs-zfs-localpv (set up for the migration) running on Talos 7.2 (K8s 1.30.1)
The goal is to migrate from storageClass openebs-zfs (Cluster A) to sa-hdd (which is a linstor sc) (Cluster B). For this I created the mapper cm as described in the docs above, and I am using the CSI snapshot driver
Both have the snapshotcontroller installed from there :https://github.com/piraeusdatastore/helm-charts/tree/main/charts/snapshot-controller which at the moment pulls v7.0.2 from upstream
Cluster A: velero backup create : success :

Click me
``` Name: www-backup Namespace: velero Labels: velero.io/storage-location=default Annotations: velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.30.1+rke2r1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=30 Phase: Completed Namespaces: Included: www Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Or label selector: Storage Location: default Velero-Native Snapshot PVs: auto Snapshot Move Data: false Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2024-05-27 11:05:26 +0200 CEST Completed: 2024-05-27 11:05:43 +0200 CEST Expiration: 2024-06-26 11:05:26 +0200 CEST Total items to be backed up: 50 Items backed up: 50 Backup Item Operations: 4 of 4 completed successfully, 0 failed (specify --details for more information) Backup Volumes: Velero-Native Snapshots: CSI Snapshots: www/data-wordpress-mariadb-0: Snapshot: included, specify --details for more information www/wordpress: Snapshot: included, specify --details for more information Pod Volume Backups: HooksAttempted: 0 HooksFailed: 0 ```
Cluster B: velero restore: success

Click me
``` Name: www-backup-20240527111555 Namespace: velero Labels: Annotations: Phase: Completed Total items to be restored: 44 Items restored: 44 Started: 2024-05-27 11:15:55 +0200 CEST Completed: 2024-05-27 11:15:59 +0200 CEST Warnings: Velero: Cluster: could not restore, CustomResourceDefinition "certificaterequests.cert-manager.io" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, CustomResourceDefinition "certificates.cert-manager.io" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, CustomResourceDefinition "orders.acme.cert-manager.io" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, CustomResourceDefinition "volumesnapshots.snapshot.storage.k8s.io" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, VolumeSnapshotClass "openebs-zfs-snapshot" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, VolumeSnapshotContent "snapcontent-400499d9-31a0-4d25-b966-3647a14f334c" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, VolumeSnapshotContent "snapcontent-cd3956d2-75bf-4e47-9d35-d5dcfe667e68" already exists. Warning: the in-cluster version is different than the backed-up version Namespaces: www: could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, Order "redacted-tls-1-2076915453" already exists. Warning: the in-cluster version is different than the backed-up version could not restore, Order "redacted-tls-2-2076915453" already exists. Warning: the in-cluster version is different than the backed-up version Backup: www-backup Namespaces: Included: all namespaces found in the backup Excluded: Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io Cluster-scoped: auto Namespace mappings: Label selector: Or label selector: Restore PVs: auto Existing Resource Policy: ItemOperationTimeout: 4h0m0s Preserve Service NodePorts: auto HooksAttempted: 0 HooksFailed: 0 Resource List: acme.cert-manager.io/v1/Order: - www/redacted-tls-1-2076915453(failed) - www/redacted--tls-2-2076915453(failed) apiextensions.k8s.io/v1/CustomResourceDefinition: - certificaterequests.cert-manager.io(failed) - certificates.cert-manager.io(failed) - orders.acme.cert-manager.io(failed) - volumesnapshots.snapshot.storage.k8s.io(failed) apps/v1/ControllerRevision: - www/wordpress-mariadb-86df68db4c(created) apps/v1/Deployment: - www/wordpress(created) apps/v1/ReplicaSet: - www/wordpress-576bd79b66(created) apps/v1/StatefulSet: - www/wordpress-mariadb(created) cert-manager.io/v1/Certificate: - www/redacted--tls(created) cert-manager.io/v1/CertificateRequest: - www/redacted--tls-1(created) - www/redacted--tls-2(created) discovery.k8s.io/v1/EndpointSlice: - www/wordpress-2p7t8(created) - www/wordpress-mariadb-9mlrq(created) networking.k8s.io/v1/Ingress: - www/wordpress(created) snapshot.storage.k8s.io/v1/VolumeSnapshot: - www/velero-data-wordpress-mariadb-0-fqsdz(created) - www/velero-wordpress-9krhg(created) - www/zfspv-snap-test(created) snapshot.storage.k8s.io/v1/VolumeSnapshotClass: - openebs-zfs-snapshot(failed) snapshot.storage.k8s.io/v1/VolumeSnapshotContent: - snapcontent-0ff773c5-836b-4496-8015-b430a3a0bacb(created) - snapcontent-400499d9-31a0-4d25-b966-3647a14f334c(failed) - snapcontent-cd3956d2-75bf-4e47-9d35-d5dcfe667e68(failed) v1/ConfigMap: - www/kube-root-ca.crt(failed) - www/wordpress-mariadb(created) v1/Endpoints: - www/wordpress(created) - www/wordpress-mariadb(created) v1/Namespace: - www(created) v1/PersistentVolume: - pvc-7c23d2a1-4e20-4bc1-ae3f-12ee07482602(skipped) - pvc-affa5d74-cac2-4d01-84eb-2cac34c0ef27(skipped) v1/PersistentVolumeClaim: - www/data-wordpress-mariadb-0(created) - www/wordpress(created) v1/Pod: - www/wordpress-576bd79b66-r4cw4(created) - www/wordpress-mariadb-0(created) v1/Secret: - www/sh.helm.release.v1.wordpress.v1(created) - www/sh.helm.release.v1.wordpress.v2(created) - www/wordpress(created) - www/wordpress-mariadb(created) - wwwredacted--tls(created) v1/Service: - www/wordpress(created) - www/wordpress-mariadb(created) v1/ServiceAccount: - www/default(skipped) - www/wordpress(created) - www/wordpress-mariadb(created) ```

However, on Cluster B the pods stay in pending because the PVs never get instantiated, on the pvc events we see :

Normal   WaitForFirstConsumer  25m (x2 over 25m)    persistentvolume-controller                                                                          waiting for first consumer to be created before binding
Warning  ProvisioningFailed    6m3s (x14 over 25m)  linstor.csi.linbit.com_linstor-csi-controller-8444d96cdc-4fd5n_8188a066-ad4d-49c0-a394-1d6d5e682708  failed to provision volume with StorageClass "sa-hdd": error getting handle for DataSource Type VolumeSnapshot by Name velero-data-wordpress-mariadb-0-fqsdz: snapshot velero-data-wordpress-mariadb-0-fqsdz not bound
Normal   Provisioning          63s (x15 over 25m)   linstor.csi.linbit.com_linstor-csi-controller-8444d96cdc-4fd5n_8188a066-ad4d-49c0-a394-1d6d5e682708  External provisioner is provisioning volume for claim "www/data-wordpress-mariadb-0"
Normal   ExternalProvisioning  14s (x103 over 25m)  persistentvolume-controller                                                                          Waiting for a volume to be created either by the external provisioner 'linstor.csi.linbit.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

What did you expect to happen:

The workload to be successfully created and bound to the migrated pvc/pv

The following information will help us better understand what's going on:

bundle-2024-05-27-11-45-47.tar.gz

Anything else you would like to add:

Cluster A: Relevant Logs of the snapshot-controller on Cluster A when running velero backup:

Click me
``` E0527 08:02:28.641756 1 snapshot_controller.go:223] error check and add Snapshot finalizers for snapshot [velero-wordpress-fkxbx]: snapshot controller failed to update www/velero-wordpress-fkxbx on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-wordpress-fkxbx": the object has been modified; please apply your changes to the latest version and try again E0527 08:02:28.641824 1 snapshot_controller_base.go:470] could not sync snapshot "www/velero-wordpress-fkxbx": snapshot controller failed to update www/velero-wordpress-fkxbx on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-wordpress-fkxbx": the object has been modified; please apply your changes to the latest version and try again I0527 08:02:28.641917 1 snapshot_controller.go:660] createSnapshotContent: Creating content for snapshot www/velero-wordpress-fkxbx through the plugin ... I0527 08:02:28.642700 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-wordpress-fkxbx", UID:"8143a8e3-8971-4805-9639-558d09c78cac", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"376222151", FieldPath:""}): type: 'Warning' reason: 'SnapshotFinalizerError' Failed to check and update snapshot: snapshot controller failed to update www/velero-wordpress-fkxbx on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-wordpress-fkxbx": the object has been modified; please apply your changes to the latest version and try again I0527 08:02:28.654529 1 snapshot_controller.go:941] Added protection finalizer to persistent volume claim www/wordpress I0527 08:02:28.665616 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-wordpress-fkxbx", UID:"8143a8e3-8971-4805-9639-558d09c78cac", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"376222152", FieldPath:""}): type: 'Normal' reason: 'CreatingSnapshot' Waiting for a snapshot www/velero-wordpress-fkxbx to be created by the CSI driver. I0527 08:02:30.632069 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-wordpress-fkxbx", UID:"8143a8e3-8971-4805-9639-558d09c78cac", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"376222179", FieldPath:""}): type: 'Normal' reason: 'SnapshotReady' Snapshot www/velero-wordpress-fkxbx is ready to use. E0527 08:02:31.035087 1 snapshot_controller_base.go:470] could not sync snapshot "www/velero-wordpress-fkxbx": snapshot controller failed to update www/velero-wordpress-fkxbx on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-wordpress-fkxbx": the object has been modified; please apply your changes to the latest version and try again I0527 08:02:31.035226 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"", Name:"", UID:"", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SnapshotStatusUpdateFailed' Snapshot status update failed, snapshot controller failed to update www/velero-wordpress-fkxbx on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-wordpress-fkxbx": the object has been modified; please apply your changes to the latest version and try again E0527 08:02:33.767820 1 snapshot_controller.go:223] error check and add Snapshot finalizers for snapshot [velero-data-wordpress-mariadb-0-cwtsz]: snapshot controller failed to update www/velero-data-wordpress-mariadb-0-cwtsz on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-data-wordpress-mariadb-0-cwtsz": the object has been modified; please apply your changes to the latest version and try again E0527 08:02:33.767895 1 snapshot_controller_base.go:470] could not sync snapshot "www/velero-data-wordpress-mariadb-0-cwtsz": snapshot controller failed to update www/velero-data-wordpress-mariadb-0-cwtsz on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-data-wordpress-mariadb-0-cwtsz": the object has been modified; please apply your changes to the latest version and try again I0527 08:02:33.768002 1 snapshot_controller.go:660] createSnapshotContent: Creating content for snapshot www/velero-data-wordpress-mariadb-0-cwtsz through the plugin ... I0527 08:02:33.768393 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-data-wordpress-mariadb-0-cwtsz", UID:"d9c6c57b-5f40-4fb7-a026-8f100653da16", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"376222217", FieldPath:""}): type: 'Warning' reason: 'SnapshotFinalizerError' Failed to check and update snapshot: snapshot controller failed to update www/velero-data-wordpress-mariadb-0-cwtsz on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-data-wordpress-mariadb-0-cwtsz": the object has been modified; please apply your changes to the latest version and try again I0527 08:02:33.777152 1 snapshot_controller.go:941] Added protection finalizer to persistent volume claim www/data-wordpress-mariadb-0 I0527 08:02:33.785423 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-data-wordpress-mariadb-0-cwtsz", UID:"d9c6c57b-5f40-4fb7-a026-8f100653da16", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"376222218", FieldPath:""}): type: 'Normal' reason: 'CreatingSnapshot' Waiting for a snapshot www/velero-data-wordpress-mariadb-0-cwtsz to be created by the CSI driver. I0527 08:02:35.758475 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-data-wordpress-mariadb-0-cwtsz", UID:"d9c6c57b-5f40-4fb7-a026-8f100653da16", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"376222244", FieldPath:""}): type: 'Normal' reason: 'SnapshotReady' Snapshot www/velero-data-wordpress-mariadb-0-cwtsz is ready to use. E0527 08:02:36.162898 1 snapshot_controller_base.go:470] could not sync snapshot "www/velero-data-wordpress-mariadb-0-cwtsz": snapshot controller failed to update www/velero-data-wordpress-mariadb-0-cwtsz on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-data-wordpress-mariadb-0-cwtsz": the object has been modified; please apply your changes to the latest version and try again I0527 08:02:36.163039 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"", Name:"", UID:"", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SnapshotStatusUpdateFailed' Snapshot status update failed, snapshot controller failed to update www/velero-data-wordpress-mariadb-0-cwtsz on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-data-wordpress-mariadb-0-cwtsz": the object has been modified; please apply your changes to the latest version and try again E0527 08:02:40.279601 1 snapshot_controller_base.go:470] could not sync snapshot "www/velero-wordpress-fkxbx": snapshot controller failed to update velero-wordpress-fkxbx on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-wordpress-fkxbx": StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshots/www/velero-wordpress-fkxbx, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 8143a8e3-8971-4805-9639-558d09c78cac, UID in object meta: E0527 08:02:41.284979 1 snapshot_controller_base.go:470] could not sync snapshot "www/velero-data-wordpress-mariadb-0-cwtsz": snapshot controller failed to update velero-data-wordpress-mariadb-0-cwtsz on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "velero-data-wordpress-mariadb-0-cwtsz": StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshots/www/velero-data-wordpress-mariadb-0-cwtsz, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d9c6c57b-5f40-4fb7-a026-8f100653da16, UID in object meta: ```

Cluster A: k get volumesnapshots/volumesnapshotconents

k get volumesnapshot -A                                                                                      
NAMESPACE   NAME              READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS          SNAPSHOTCONTENT                                    CREATIONTIME   AGE
www         zfspv-snap-test   true         wordpress 

k get volumesnapshotcontents.snapshot.storage.k8s.io                                                         
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER               VOLUMESNAPSHOTCLASS    VOLUMESNAPSHOT                              VOLUMESNAPSHOTNAMESPACE                   AGE
snapcontent-0ff773c5-836b-4496-8015-b430a3a0bacb   true         10737418240   Delete           zfs.csi.openebs.io   openebs-zfs-snapshot   zfspv-snap-test                             www                                       62m
snapcontent-400499d9-31a0-4d25-b966-3647a14f334c   true         0             Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   name-2d5f0627-902b-4d8a-b857-0a62fb1da36a   ns-2d5f0627-902b-4d8a-b857-0a62fb1da36a   44m
snapcontent-cd3956d2-75bf-4e47-9d35-d5dcfe667e68   true         0             Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   name-dd02fa04-84ad-4174-8532-3e99306a96f9   ns-dd02fa04-84ad-4174-8532-3e99306a96f9   44m

NB: I created the zfspv-snap-test manually, by creating a volumesnapshot to test before running the velero commands. It worked (the snapshot exists in the zfs list... on the node), but the controller still returned the same errors as above. Note the RESTORESIZE field

Cluster B: Relevant Logs of the snapshot-controller on Cluster B when running velero restore:

Click me
``` I0526 19:02:51.950130 1 snapshot_controller.go:660] createSnapshotContent: Creating content for snapshot www/zfspv-snap-test through the plugin ... I0526 19:02:51.965987 1 snapshot_controller.go:941] Added protection finalizer to persistent volume claim www/wordpress E0526 19:02:51.966021 1 snapshot_controller.go:779] getCreateSnapshotInput failed to get PersistentVolume object [zfspv-snap-test]: Error: [&errors.errorString{s:"the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot"}] E0526 19:02:51.971025 1 snapshot_controller_base.go:470] could not sync snapshot "www/zfspv-snap-test": failed to get input parameters to create snapshot zfspv-snap-test: "the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot" I0526 19:02:51.971255 1 snapshot_controller.go:660] createSnapshotContent: Creating content for snapshot www/zfspv-snap-test through the plugin ... I0526 19:02:51.971280 1 snapshot_controller.go:925] Protection finalizer already exists for persistent volume claim www/wordpress E0526 19:02:51.971291 1 snapshot_controller.go:779] getCreateSnapshotInput failed to get PersistentVolume object [zfspv-snap-test]: Error: [&errors.errorString{s:"the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot"}] I0526 19:02:51.971698 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"", Name:"", UID:"", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SnapshotContentCreationFailed' Failed to create snapshot content with error failed to get input parameters to create snapshot zfspv-snap-test: "the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot" E0526 19:02:51.979860 1 snapshot_controller_base.go:470] could not sync snapshot "www/zfspv-snap-test": failed to get input parameters to create snapshot zfspv-snap-test: "the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot" I0526 19:02:51.979918 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"zfspv-snap-test", UID:"a960ea68-7e59-4393-945d-b0e7d153c856", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"59772", FieldPath:""}): type: 'Warning' reason: 'SnapshotContentCreationFailed' Failed to create snapshot content with error failed to get input parameters to create snapshot zfspv-snap-test: "the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot" I0526 19:02:51.980436 1 snapshot_controller.go:660] createSnapshotContent: Creating content for snapshot www/zfspv-snap-test through the plugin ... I0526 19:02:51.980455 1 snapshot_controller.go:925] Protection finalizer already exists for persistent volume claim www/wordpress E0526 19:02:51.980467 1 snapshot_controller.go:779] getCreateSnapshotInput failed to get PersistentVolume object [zfspv-snap-test]: Error: [&errors.errorString{s:"the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot"}] E0526 19:02:51.980491 1 snapshot_controller_base.go:470] could not sync snapshot "www/zfspv-snap-test": failed to get input parameters to create snapshot zfspv-snap-test: "the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot" I0526 19:02:52.971670 1 snapshot_controller.go:660] createSnapshotContent: Creating content for snapshot www/zfspv-snap-test through the plugin ... I0526 19:02:52.971714 1 snapshot_controller.go:925] Protection finalizer already exists for persistent volume claim www/wordpress E0526 19:02:52.971725 1 snapshot_controller.go:779] getCreateSnapshotInput failed to get PersistentVolume object [zfspv-snap-test]: Error: [&errors.errorString{s:"the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot"}] E0526 19:02:52.971755 1 snapshot_controller_base.go:470] could not sync snapshot "www/zfspv-snap-test": failed to get input parameters to create snapshot zfspv-snap-test: "the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot" I0526 19:03:00.972345 1 snapshot_controller.go:660] createSnapshotContent: Creating content for snapshot www/zfspv-snap-test through the plugin ... I0526 19:03:00.972369 1 snapshot_controller.go:925] Protection finalizer already exists for persistent volume claim www/wordpress E0526 19:03:00.972378 1 snapshot_controller.go:779] getCreateSnapshotInput failed to get PersistentVolume object [zfspv-snap-test]: Error: [&errors.errorString{s:"the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot"}] E0526 19:03:00.972401 1 snapshot_controller_base.go:470] could not sync snapshot "www/zfspv-snap-test": failed to get input parameters to create snapshot zfspv-snap-test: "the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot" I0526 19:03:16.972750 1 snapshot_controller.go:660] createSnapshotContent: Creating content for snapshot www/zfspv-snap-test through the plugin ... I0526 19:03:16.972778 1 snapshot_controller.go:925] Protection finalizer already exists for persistent volume claim www/wordpress E0526 19:03:16.972787 1 snapshot_controller.go:779] getCreateSnapshotInput failed to get PersistentVolume object [zfspv-snap-test]: Error: [&errors.errorString{s:"the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot"}] E0526 19:03:16.972812 1 snapshot_controller_base.go:470] could not sync snapshot "www/zfspv-snap-test": failed to get input parameters to create snapshot zfspv-snap-test: "the PVC wordpress is not yet bound to a PV, will not attempt to take a snapshot" I0526 19:03:30.817148 1 snapshot_controller.go:1020] checkandRemovePVCFinalizer[zfspv-snap-test]: Remove Finalizer for PVC wordpress as it is not used by snapshots in creation I0526 19:03:48.792116 1 snapshot_controller.go:899] isVolumeBeingCreatedFromSnapshot: volume data-wordpress-mariadb-0 is being created from snapshot velero-data-wordpress-mariadb-0-txd4s I0526 19:03:48.792254 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-data-wordpress-mariadb-0-txd4s", UID:"3e27b8fd-6a60-42d9-89f3-ba448adebf54", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"60350", FieldPath:""}): type: 'Warning' reason: 'SnapshotDeletePending' Snapshot is being used to restore a PVC I0526 19:03:48.801224 1 snapshot_controller.go:899] isVolumeBeingCreatedFromSnapshot: volume wordpress is being created from snapshot velero-wordpress-cj7tn I0526 19:03:48.801449 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-wordpress-cj7tn", UID:"d807c56e-4bb9-4a82-ac59-c065cfd99284", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"60351", FieldPath:""}): type: 'Warning' reason: 'SnapshotDeletePending' Snapshot is being used to restore a PVC I0527 09:16:02.330565 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-data-wordpress-mariadb-0-fqsdz", UID:"612cc3fd-380e-4582-b46d-bcc670aa3f2f", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"556046", FieldPath:""}): type: 'Normal' reason: 'SnapshotCreated' Snapshot www/velero-data-wordpress-mariadb-0-fqsdz was successfully created by the CSI driver. I0527 09:16:02.330631 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-data-wordpress-mariadb-0-fqsdz", UID:"612cc3fd-380e-4582-b46d-bcc670aa3f2f", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"556046", FieldPath:""}): type: 'Normal' reason: 'SnapshotReady' Snapshot www/velero-data-wordpress-mariadb-0-fqsdz is ready to use. I0527 09:16:02.730650 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-wordpress-9krhg", UID:"5e39e4f8-7d13-4310-9526-5ccae9781e1a", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"556052", FieldPath:""}): type: 'Normal' reason: 'SnapshotCreated' Snapshot www/velero-wordpress-9krhg was successfully created by the CSI driver. I0527 09:16:02.730784 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"velero-wordpress-9krhg", UID:"5e39e4f8-7d13-4310-9526-5ccae9781e1a", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"556052", FieldPath:""}): type: 'Normal' reason: 'SnapshotReady' Snapshot www/velero-wordpress-9krhg is ready to use. I0527 09:16:02.930426 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"zfspv-snap-test", UID:"e1c75fca-a5bc-4d4b-9424-839ea8cd2abe", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"556061", FieldPath:""}): type: 'Normal' reason: 'SnapshotCreated' Snapshot www/zfspv-snap-test was successfully created by the CSI driver. I0527 09:16:02.930448 1 event.go:364] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"www", Name:"zfspv-snap-test", UID:"e1c75fca-a5bc-4d4b-9424-839ea8cd2abe", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"556061", FieldPath:""}): type: 'Normal' reason: 'SnapshotReady' Snapshot www/zfspv-snap-test is ready to use. ```

Cluster B: k get volumesnapshots/volumesnapshotconents

k get volumesnapshot -A                                                                            
NAMESPACE   NAME                                    READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT                                RESTORESIZE   SNAPSHOTCLASS          SNAPSHOTCONTENT                                      CREATIONTIME   AGE
www         velero-data-wordpress-mariadb-0-fqsdz   true                     velero-velero-data-wordpress-mariadb-0-fqsdz-kq22m   0             openebs-zfs-snapshot   velero-velero-data-wordpress-mariadb-0-fqsdz-kq22m   43m            43m
www         velero-wordpress-9krhg                  true                     velero-velero-wordpress-9krhg-nvtn9                  0             openebs-zfs-snapshot   velero-velero-wordpress-9krhg-nvtn9                  43m            43m
www         zfspv-snap-test                         true                     velero-zfspv-snap-test-jp7j6                         0             openebs-zfs-snapshot   velero-zfspv-snap-test-jp7j6                         43m            43m

k get volumesnapshotcontents.snapshot.storage.k8s.io -A                                            
NAME                                                 READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER               VOLUMESNAPSHOTCLASS    VOLUMESNAPSHOT                          VOLUMESNAPSHOTNAMESPACE   AGE
snapcontent-0ff773c5-836b-4496-8015-b430a3a0bacb                                Delete           zfs.csi.openebs.io   openebs-zfs-snapshot   zfspv-snap-test                         www                       43m
snapcontent-2f02e2b3-2fcd-4cf8-bbf2-211de1517a1c                                Delete           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-wordpress-jg9xt                  www                       14h
snapcontent-400499d9-31a0-4d25-b966-3647a14f334c                                Delete           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-data-wordpress-mariadb-0-fqsdz   www                       53m
snapcontent-8143a8e3-8971-4805-9639-558d09c78cac                                Delete           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-wordpress-fkxbx                  www                       116m
snapcontent-94a02015-0a96-43e0-b46c-939a3dfed4e8                                Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-wordpress-cj7tn                  www                       16h
snapcontent-98eb647a-03d3-4611-8eff-7653921547b3                                Delete           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-data-wordpress-mariadb-0-qncvq   www                       14h
snapcontent-c4ff5bd0-e791-421c-a357-2daf95ae97d1                                Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-data-wordpress-mariadb-0-txd4s   www                       16h
snapcontent-cd3956d2-75bf-4e47-9d35-d5dcfe667e68                                Delete           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-wordpress-9krhg                  www                       53m
snapcontent-d5442acb-e6a0-42d1-b790-30d29a2d647b                                Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-data-wordpress-mariadb-0-mp75g   www                       15h
snapcontent-d9c6c57b-5f40-4fb7-a026-8f100653da16                                Delete           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-data-wordpress-mariadb-0-cwtsz   www                       116m
snapcontent-ef022ffe-d9d6-459f-9c83-bd0012fe1b3b                                Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-wordpress-279sg                  www                       15h
velero-velero-data-wordpress-mariadb-0-fqsdz-kq22m   true         0             Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-data-wordpress-mariadb-0-fqsdz   www                       43m
velero-velero-data-wordpress-mariadb-0-txd4s-v698g   true         0             Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-data-wordpress-mariadb-0-txd4s   www                       16h
velero-velero-wordpress-9krhg-nvtn9                  true         0             Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-wordpress-9krhg                  www                       43m
velero-velero-wordpress-cj7tn-5srn4                  true         0             Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   velero-wordpress-cj7tn                  www                       16h
velero-zfspv-snap-test-jp7j6                         true         0             Retain           zfs.csi.openebs.io   openebs-zfs-snapshot   zfspv-snap-test                         www                       43m

NB: you can see some previous attempt to restore there. see the AGE column

I installed velero using the following command

velero install \
              --features=EnableCSI,EnableAPIGroupVersions \
              --use-volume-snapshots \
              --plugins=velero/velero-plugin-for-aws:$aws_plugin_vers,velero/velero-plugin-for-csi:$csi_plugin_vers \
              --provider aws --bucket $aws_bucket --secret-file s3-home.secret \
              --backup-location-config region=$aws_region,s3ForcePathStyle="true",s3Url=$S3URL \
              --snapshot-location-config region=$aws_region

Environment:

Velero version (use velero version):

Client:
  Version: v1.13.2
  Git commit: -
Server:
  Version: v1.13.2

Velero features (use velero client config get features): features: EnableCSI,EnableAPIGroupVersions

Kubernetes version (use kubectl version):

# cluster A
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1+rke2r1

# cluster B
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1

Kubernetes installer & version: RKE2 (ClusterA), Talos 1.7.2 (ClusterB)
Cloud provider or hardware configuration: N/A. Both runnning on bare-metal servers
OS (e.g. from /etc/os-release): see above

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

:+1: for "I would like to see this bug fixed as soon as possible"
:-1: for "There are more important bugs to focus on right now"

vmware-tanzu / velero

Migration between clusters not working as expected: snapshot not bound #7832