search

IBM / ibm-spectrum-scale-csi

The IBM Spectrum Scale Container Storage Interface (CSI) project enables container orchestrators, such as Kubernetes and OpenShift, to manage the life-cycle of persistent storage.
Apache License 2.0
66 stars 49 forks source link

shallow copy pv doesn't get deleted if snapshot directory gets deleted #1080

Closed saurabhwani5 closed 7 months ago

saurabhwani5 commented 9 months ago

Describe the bug

When snapshot dir gets deleted accidentally,shallow copy volume doesn't get deleted with reason : rpc error: code = Unknown desc = unable to stat dir pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed/snapshot-5bba074e-da41-4901-8aa6-f1f78db36fa4:[EFSSG0264C The path /var/gpfs/fs0/pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed/snapshot-5bba074e-da41-4901-8aa6-f1f78db36fa4 does not exist.]

How to Reproduce?

  1. Install CSI 2.11.0 with #1067 driver images: 
    [root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc get pods
NAME                                                  READY   STATUS    RESTARTS      AGE
ibm-spectrum-scale-csi-5f9vx                          3/3     Running   0             4d23h
ibm-spectrum-scale-csi-attacher-585ddbd6bb-l4n62      1/1     Running   0             4d23h
ibm-spectrum-scale-csi-attacher-585ddbd6bb-mc5bk      1/1     Running   0             4d23h
ibm-spectrum-scale-csi-f69cw                          3/3     Running   0             4d23h
ibm-spectrum-scale-csi-knlvd                          3/3     Running   0             4d23h
ibm-spectrum-scale-csi-operator-5c6565c84c-q4c7p      1/1     Running   0             4d23h
ibm-spectrum-scale-csi-provisioner-765d779f6b-gjkhc   1/1     Running   0             4d23h
ibm-spectrum-scale-csi-resizer-644c7bd59b-8rnlj       1/1     Running   0             4d23h
ibm-spectrum-scale-csi-snapshotter-7cd97548d5-j8stg   1/1     Running   2 (37h ago)   4d23h
[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.11.0    True
[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc describe pod | grep quay
Image:         quay.io/hemalatha_gajendran/driver_shallowcopy_withoutfileset
Image ID:      quay.io/hemalatha_gajendran/driver_shallowcopy_withoutfileset@sha256:8f1d9b7d5c80ca2b7c173584ca97d2587143f8e84d5c7bfa48d3618e43e9119b
Image:         quay.io/hemalatha_gajendran/driver_shallowcopy_withoutfileset
Image ID:      quay.io/hemalatha_gajendran/driver_shallowcopy_withoutfileset@sha256:8f1d9b7d5c80ca2b7c173584ca97d2587143f8e84d5c7bfa48d3618e43e9119b
Image:         quay.io/hemalatha_gajendran/driver_shallowcopy_withoutfileset
Image ID:      quay.io/hemalatha_gajendran/driver_shallowcopy_withoutfileset@sha256:8f1d9b7d5c80ca2b7c173584ca97d2587143f8e84d5c7bfa48d3618e43e9119b
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:18264e0c9c112856bc2744f7f971a4b60ecb24de57e46af4d35456dcdf8e3cbf
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:18264e0c9c112856bc2744f7f971a4b60ecb24de57e46af4d35456dcdf8e3cbf
  CSI_DRIVER_IMAGE:          quay.io/hemalatha_gajendran/driver_shallowcopy_withoutfileset
  2. Create PVC and pod as following : 
    
[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# cat apply.yaml
apiVersion: v1
kind: Pod
metadata:
name: csi-scale-fsetdemo-pod-1
labels:
app: nginx
spec:
containers:
- name: web-server
 image: docker-na-public.artifactory.swg-devops.com/sys-spectrum-scale-team-test-environment-docker-local/nginx:1.22.0
 volumeMounts:
   - name: mypvc
     mountPath: /usr/share/nginx/html/scale
 ports:
 - containerPort: 80
volumes:
- name: mypvc
 persistentVolumeClaim:
   claimName: scale-pvc-1
   readOnly: false

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scale-pvc-1 spec: accessModes:

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ibm-spectrum-scale-csi provisioner: spectrumscale.csi.ibm.com parameters: volBackendFs: "fs0" reclaimPolicy: Delete

[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE scale-pvc-1 Bound pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed 1Gi RWX ibm-spectrum-scale-csi 3m48s [root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# [root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc get pods NAME READY STATUS RESTARTS AGE csi-scale-fsetdemo-pod-1 1/1 Running 0 3m52s

3. Write data in pod inside mount path:

root@csi-scale-fsetdemo-pod-1:/# cd /usr/share/nginx/html/scale root@csi-scale-fsetdemo-pod-1:/usr/share/nginx/html/scale# touch test{1..10} root@csi-scale-fsetdemo-pod-1:/usr/share/nginx/html/scale# ls test1 test10 test2 test3 test4 test5 test6 test7 test8 test9

4. Take snapshot of above pvc:

[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# cat snapshot.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: ibm-spectrum-scale-snapshot spec: volumeSnapshotClassName: ibm-spectrum-scale-snapshotclass source: persistentVolumeClaimName: scale-pvc-1

apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: ibm-spectrum-scale-snapshotclass driver: spectrumscale.csi.ibm.com parameters: snapWindow: "30" #Optional : Time in minutes (default=30) deletionPolicy: Delete

[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc apply -f snapshot.yaml volumesnapshot.snapshot.storage.k8s.io/ibm-spectrum-scale-snapshot created volumesnapshotclass.snapshot.storage.k8s.io/ibm-spectrum-scale-snapshotclass created

5. create shalllow copy from above snapshot:

[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc get vs -w NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE ibm-spectrum-scale-snapshot false scale-pvc-1 ibm-spectrum-scale-snapshotclass snapcontent-5bba074e-da41-4901-8aa6-f1f78db36fa4 16s ibm-spectrum-scale-snapshot true scale-pvc-1 1Gi ibm-spectrum-scale-snapshotclass snapcontent-5bba074e-da41-4901-8aa6-f1f78db36fa4 10s 24s

[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# cat shallowCopy.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scale-shallow-copy-pvc-1 spec: accessModes:

[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc apply -f shallowCopy.yaml persistentvolumeclaim/scale-shallow-copy-pvc-1 created [root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE scale-pvc-1 Bound pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed 1Gi RWX ibm-spectrum-scale-csi 13m scale-shallow-copy-pvc-1 Bound pvc-76c07f21-9663-4612-a5af-d0ebe7d012ba 1Gi ROX ibm-spectrum-scale-csi 49s

6. Delete the snapshot directory from the fileset of source volume:

[root@remote-saurabhwani1213-2 pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed]# ls pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed-data snapshot-5bba074e-da41-4901-8aa6-f1f78db36fa4 [root@remote-saurabhwani1213-2 pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed]# rm -rf snapshot-5bba074e-da41-4901-8aa6-f1f78db36fa4/


7. delete the shallow copy pvc and check if shallow copy is getting deleted or not :

[root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc delete pvc scale-shallow-copy-pvc-1 persistentvolumeclaim "scale-shallow-copy-pvc-1" deleted [root@api.cnsa-saurabhwani1213.cp.fyre.ibm.com Upgradetesting]# oc describe pv pvc-76c07f21-9663-4612-a5af-d0ebe7d012ba Name: pvc-76c07f21-9663-4612-a5af-d0ebe7d012ba Labels: Annotations: pv.kubernetes.io/provisioned-by: spectrumscale.csi.ibm.com volume.kubernetes.io/provisioner-deletion-secret-name: volume.kubernetes.io/provisioner-deletion-secret-namespace: Finalizers: [kubernetes.io/pv-protection external-attacher/spectrumscale-csi-ibm-com] StorageClass: ibm-spectrum-scale-csi Status: Released Claim: ibm-spectrum-scale-csi/scale-shallow-copy-pvc-1 Reclaim Policy: Delete Access Modes: ROX VolumeMode: Filesystem Capacity: 1Gi Node Affinity: Message: Source: Type: CSI (a Container Storage Interface (CSI) volume source) Driver: spectrumscale.csi.ibm.com FSType: gpfs VolumeHandle: 0;3;4181049054023231843;263B0B0A:6589AA52;;pvc-76c07f21-9663-4612-a5af-d0ebe7d012ba;/mnt/fs0/pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed/.snapshots/snapshot-5bba074e-da41-4901-8aa6-f1f78db36fa4/pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed-data ReadOnly: false VolumeAttributes: csi.storage.k8s.io/pv/name=pvc-76c07f21-9663-4612-a5af-d0ebe7d012ba csi.storage.k8s.io/pvc/name=scale-shallow-copy-pvc-1 csi.storage.k8s.io/pvc/namespace=ibm-spectrum-scale-csi storage.kubernetes.io/csiProvisionerIdentity=1703587394100-8615-spectrumscale.csi.ibm.com volBackendFs=fs0 Events: Type Reason Age From Message

Warning VolumeFailedDelete 58s (x2 over 59s) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-765d779f6b-gjkhc_b983f2f8-f0f3-4464-a094-f633e7391e29 persistentvolume pvc-76c07f21-9663-4612-a5af-d0ebe7d012ba is still attached to node worker2.cnsa-saurabhwani1213.cp.fyre.ibm.com Warning VolumeFailedDelete 18s (x3 over 47s) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-765d779f6b-gjkhc_b983f2f8-f0f3-4464-a094-f633e7391e29 rpc error: code = Unknown desc = unable to stat dir pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed/snapshot-5bba074e-da41-4901-8aa6-f1f78db36fa4:[EFSSG0264C The path /var/gpfs/fs0/pvc-afb220f3-dfd2-43d9-8031-d6c9de8c5aed/snapshot-5bba074e-da41-4901-8aa6-f1f78db36fa4 does not exist.]


## Expected behavior
shallow copy volume should be deleted if snapshot directory is not present 

## Logs

/scale-csi/D.1080 csisnap.tar.gz

saurabhwani5 commented 8 months ago

This issue is also seen in type of issue in good path scenario Warning VolumeFailedDelete 25s (x167 over 11h) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-6dd44f9b4-pvvvd_3232e8a8-8dad-4f65-a291-b90c5575e30c rpc error: code = Unknown desc = unable to stat dir pvc-d80c2ec5-35c5-4eeb-9893-a45ac0e8645e/snapshot-vheldksd-0:[EFSSG0264C The path /ibm/fs1/pvc-d80c2ec5-35c5-4eeb-9893-a45ac0e8645e/snapshot-vheldksd-0 does not exist.]

saurabhwani5 commented 7 months ago

closing as verified , thanks @hemalathagajendran