shallow copy volume is not getting created with old snapshot handle

Describe the bug

when we try to create shallow copy volume with old snapshot handle then shallow copy does not get created with error message CreateVolume ValidateShallowCopyVolume failed

How to Reproduce?

Install CSI 2.4.0:

root@saurabhubu22-master:~# oc get pods
NAME                                               READY   STATUS    RESTARTS       AGE
csi-scale-fsetdemo-pod-2                           1/1     Running   0              5d2h
ibm-spectrum-scale-csi-4l7z8                       3/3     Running   3 (9m6s ago)   75m
ibm-spectrum-scale-csi-attacher-0                  1/1     Running   3 (9m6s ago)   75m
ibm-spectrum-scale-csi-mlxvs                       3/3     Running   3 (13m ago)    75m
ibm-spectrum-scale-csi-operator-559dd46bd4-6nmqp   1/1     Running   0              78m
ibm-spectrum-scale-csi-provisioner-0               1/1     Running   3 (9m6s ago)   75m
ibm-spectrum-scale-csi-resizer-0                   1/1     Running   3 (9m6s ago)   75m
ibm-spectrum-scale-csi-snapshotter-0               1/1     Running   3 (13m ago)    75m
root@saurabhubu22-master:~# oc describe pod | grep quay
Image:         quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver:v2.4.0
Image ID:      quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver@sha256:4d8c41138f2fddac351f82db19c32fe5ad1282e7886f78fe2669f0c30ea5badb
Image:         quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver:v2.4.0
Image ID:      quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver@sha256:4d8c41138f2fddac351f82db19c32fe5ad1282e7886f78fe2669f0c30ea5badb
Image:         quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.4.0
Image ID:      quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator@sha256:38751e2b7a4624e588747ed427c2c2146bee320bab74b0bb288f38c2c5d2bddd
  CSI_DRIVER_IMAGE:                                        quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver:v2.4.0

Create Version 1 Independent PVC :


root@saurabhubu22-master:~/100-volumes/Independent# cat sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibm-spectrum-scale-csi-fileset-independent
provisioner: spectrumscale.csi.ibm.com
parameters:
volBackendFs: "fs1"
reclaimPolicy: Deleteroot@saurabhubu22-master:~/100-volumes/Independent# cat pvc_independent_1.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scale-fset-independent-pvc-1
spec:
accessModes:
- ReadWriteMany
resources:
requests:
  storage: 1Gi

root@saurabhubu22-master:~/100-volumes/Independent# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE scale-fset-independent-pvc-1 Bound pvc-6f1eee30-5cfe-4e65-9d77-9aba8f60af1f 1Gi RWX ibm-spectrum-scale-csi-fileset-independent 70m

3. Create snapshot of the same:

root@saurabhubu22-master:~/100-volumes/Independent# cat vsc.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: ibm-spectrum-scale-snapshotclass driver: spectrumscale.csi.ibm.com deletionPolicy: Delete root@saurabhubu22-master:~/100-volumes/Independent# cat vs.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: ibm-spectrum-scale-snapshot spec: volumeSnapshotClassName: ibm-spectrum-scale-snapshotclass source: persistentVolumeClaimName: scale-fset-independent-pvc-1

4. Check the description of snapshot content :

root@saurabhubu22-master:~/100-volumes/Independent# oc describe vs ibm-spectrum-scale-snapshot Name: ibm-spectrum-scale-snapshot Namespace: ibm-spectrum-scale-csi-driver Labels: Annotations: API Version: snapshot.storage.k8s.io/v1 Kind: VolumeSnapshot Metadata: Creation Timestamp: 2024-01-30T07:28:04Z Finalizers: snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection Generation: 1 Resource Version: 1687409 UID: 286f1eea-e7ba-422d-8fc5-8af64b8d634b Spec: Source: Persistent Volume Claim Name: scale-fset-independent-pvc-1 Volume Snapshot Class Name: ibm-spectrum-scale-snapshotclass Status: Bound Volume Snapshot Content Name: snapcontent-286f1eea-e7ba-422d-8fc5-8af64b8d634b Creation Time: 2024-01-30T07:28:38Z Ready To Use: true Restore Size: 1Gi Events: root@saurabhubu22-master:~/100-volumes/Independent# oc describe volumesnapshotcontent snapcontent-286f1eea-e7ba-422d-8fc5-8af64b8d634b Name: snapcontent-286f1eea-e7ba-422d-8fc5-8af64b8d634b Namespace: Labels: Annotations: API Version: snapshot.storage.k8s.io/v1 Kind: VolumeSnapshotContent Metadata: Creation Timestamp: 2024-01-30T07:28:04Z Finalizers: snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection Generation: 1 Resource Version: 1687407 UID: f2889bef-a57f-46a0-9da6-48a056008159 Spec: Deletion Policy: Delete Driver: spectrumscale.csi.ibm.com Source: Volume Handle: 15590680369348351161;8D0E0B0A:65AA15A1;filesetName=pvc-6f1eee30-5cfe-4e65-9d77-9aba8f60af1f;path=/ibm/fs1/spectrum-scale-csi-volume-store/.volumes/pvc-6f1eee30-5cfe-4e65-9d77-9aba8f60af1f Volume Snapshot Class Name: ibm-spectrum-scale-snapshotclass Volume Snapshot Ref: API Version: snapshot.storage.k8s.io/v1 Kind: VolumeSnapshot Name: ibm-spectrum-scale-snapshot Namespace: ibm-spectrum-scale-csi-driver Resource Version: 1687318 UID: 286f1eea-e7ba-422d-8fc5-8af64b8d634b Status: Creation Time: 1706599718742153534 Ready To Use: true Restore Size: 1073741824

5. Upgrade to CSI 2.11.0 dev :

root@saurabhubu22-master:~/2.11# oc apply -f ibm-spectrum-scale-csi-operator.yaml deployment.apps/ibm-spectrum-scale-csi-operator configured clusterrole.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator configured clusterrolebinding.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator unchanged serviceaccount/ibm-spectrum-scale-csi-operator configured customresourcedefinition.apiextensions.k8s.io/csiscaleoperators.csi.ibm.com configured root@saurabhubu22-master:~/2.11# oc get cso -w NAME VERSION SUCCESS ibm-spectrum-scale-csi ibm-spectrum-scale-csi 2.11.0 False ibm-spectrum-scale-csi 2.11.0 False ibm-spectrum-scale-csi 2.11.0 True root@saurabhubu22-master:~/100-volumes/Independent# oc get pods NAME READY STATUS RESTARTS AGE csi-scale-fsetdemo-pod-2 1/1 Running 0 5d3h ibm-spectrum-scale-csi-attacher-869bd7ff6d-blv4l 1/1 Running 0 4m54s ibm-spectrum-scale-csi-attacher-869bd7ff6d-fbw5h 1/1 Running 1 (4m49s ago) 4m54s ibm-spectrum-scale-csi-nq9sf 3/3 Running 0 4m53s ibm-spectrum-scale-csi-operator-7fc87756c9-lsvlp 1/1 Running 0 5m13s ibm-spectrum-scale-csi-pq9qz 3/3 Running 0 4m49s ibm-spectrum-scale-csi-provisioner-c48d8df47-tjb8c 1/1 Running 0 4m54s ibm-spectrum-scale-csi-resizer-54c67667c4-bkdpj 1/1 Running 0 4m54s ibm-spectrum-scale-csi-snapshotter-6f4964bd9b-rjxv6 1/1 Running 0 4m54s root@saurabhubu22-master:~/100-volumes/Independent# oc describe pod | grep quay Image: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:87ba864fb132882215fc7322df2644e8a9f7d47e2471c9f108b0c72bb82adb24 Image ID: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:33e094c5548233a718e8c882e4799e6dd5adaabe31e017f1da91219450292399 Normal Pulled 4m59s kubelet Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:87ba864fb132882215fc7322df2644e8a9f7d47e2471c9f108b0c72bb82adb24" already present on machine Image: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:942a9a27438ca7458ae3c5f02ec3bb9e537c7ce7fc914a36afce46529901a785 Image ID: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:942a9a27438ca7458ae3c5f02ec3bb9e537c7ce7fc914a36afce46529901a785 CSI_DRIVER_IMAGE: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:87ba864fb132882215fc7322df2644e8a9f7d47e2471c9f108b0c72bb82adb24 Normal Pulled 5m19s kubelet Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:942a9a27438ca7458ae3c5f02ec3bb9e537c7ce7fc914a36afce46529901a785" already present on machine Image: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:87ba864fb132882215fc7322df2644e8a9f7d47e2471c9f108b0c72bb82adb24 Image ID: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:33e094c5548233a718e8c882e4799e6dd5adaabe31e017f1da91219450292399 Normal Pulled 4m55s kubelet Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:87ba864fb132882215fc7322df2644e8a9f7d47e2471c9f108b0c72bb82adb24" already present on machine

6. Create shallow copy from snapshot as following :

root@saurabhubu22-master:~/100-volumes/Independent# cat restore.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ibm-spectrum-scale-pvc-shallow-copy spec: accessModes:

ReadOnlyMany resources: requests: storage: 1Gi storageClassName: ibm-spectrum-scale-csi-fileset-independent dataSource: name: ibm-spectrum-scale-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io root@saurabhubu22-master:~/100-volumes/Independent# oc apply -f restore.yaml persistentvolumeclaim/ibm-spectrum-scale-pvc-shallow-copy created
```
7. Check pvc description:
```
root@saurabhubu22-master:~/100-volumes/Independent# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ibm-spectrum-scale-pvc-shallow-copy Pending ibm-spectrum-scale-csi-fileset-independent 63s scale-fset-independent-pvc-1 Bound pvc-6f1eee30-5cfe-4e65-9d77-9aba8f60af1f 1Gi RWX ibm-spectrum-scale-csi-fileset-independent 85m root@saurabhubu22-master:~/100-volumes/Independent# oc describe pvc ibm-spectrum-scale-pvc-shallow-copy Name: ibm-spectrum-scale-pvc-shallow-copy Namespace: ibm-spectrum-scale-csi-driver StorageClass: ibm-spectrum-scale-csi-fileset-independent Status: Pending Volume: Labels: Annotations: volume.beta.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com volume.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem DataSource: APIGroup: snapshot.storage.k8s.io Kind: VolumeSnapshot Name: ibm-spectrum-scale-snapshot Used By: Events: Type Reason Age From Message

Warning ProvisioningFailed 12s spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-c48d8df47-tjb8c_f5f4fc12-20c1-478d-8310-047475bbd621 failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset-independent": rpc error: code = Internal desc = CreateVolume ValidateShallowCopyVolume failed Normal ExternalProvisioning 11s (x6 over 68s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "spectrumscale.csi.ibm.com" or manually created by system administrator Normal Provisioning 11s (x2 over 27s) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-c48d8df47-tjb8c_f5f4fc12-20c1-478d-8310-047475bbd621 External provisioner is provisioning volume for claim "ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-pvc-shallow-copy"

Expected behavior

Shallow copy volume should get created with old snapshot

Logs

/scale-csi/D.1090
csisnap.tar.gz

IBM / ibm-spectrum-scale-csi