search

IBM / ibm-spectrum-scale-csi

The IBM Spectrum Scale Container Storage Interface (CSI) project enables container orchestrators, such as Kubernetes and OpenShift, to manage the life-cycle of persistent storage.
Apache License 2.0
64 stars 50 forks source link

PV remains in terminating state with provisioner sidecar v5.0.1 #1173

Closed shrutinipane closed 1 week ago

shrutinipane commented 3 weeks ago

Describe the bug

PV remains in terminating state when we try to delete a lightweight volume provisioned by provisioner sidecar version v5.0.1 (registry.k8s.io/sig-storage/csi-provisioner@sha256:405a14e1aa702f7ea133cea459e8395fe40a6125c088c55569e696d48e1bd385).

How to Reproduce?

  1. Apply Operator image: quay.io/shrutika_nipane/ibm-spectrum-scale-test:sidecars2
  2. Create lightweight volume 
    
[root@csi-kube129-1-x-master lightweight]# cat storageclasslw.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibm-spectrum-scale-csi-lt
provisioner: spectrumscale.csi.ibm.com
parameters:
volBackendFs: "fs1"
volDirBasePath: "LW"
reclaimPolicy: Delete

[root@csi-kube129-1-x-master lightweight]# [root@csi-kube129-1-x-master lightweight]# oc apply -f storageclasslw.yaml storageclass.storage.k8s.io/ibm-spectrum-scale-csi-lt created

3. Create PVC using lightweight volume

[root@csi-kube129-1-x-master lightweight]# cat pvclw.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scale-lt-pvc spec: accessModes:

[root@csi-kube129-1-x-master lightweight]# oc apply -f pvclw.yaml persistentvolumeclaim/scale-lt-pvc created [root@csi-kube129-1-x-master lightweight]# oc get pvc -w NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE scale-lt-pvc Pending ibm-spectrum-scale-csi-lt 20s scale-lt-pvc Pending pvc-261406a1-923b-4cc0-bad4-bebf037acf94 0 ibm-spectrum-scale-csi-lt 31s scale-lt-pvc Bound pvc-261406a1-923b-4cc0-bad4-bebf037acf94 1Gi RWX ibm-spectrum-scale-csi-lt 31s

4. Delete pvc

## Expected behavior
PV should be deleted after PVC deletion but it remains in Terminating state.

[root@csi-kube129-1-x-master lightweight]# oc get pv pvc-261406a1-923b-4cc0-bad4-bebf037acf94 -w NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE pvc-261406a1-923b-4cc0-bad4-bebf037acf94 1Gi RWX Delete Terminating ibm-spectrum-scale-csi-driver/scale-lt-pvc ibm-spectrum-scale-csi-lt 

Events in PV:

Events: Type Reason Age From Message

Warning VolumeFailedDelete 6s (x6 over 103s) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-ff75d95c-5dh4l_03fc7cd8-3c7a-4c75-b3a0-42117bea5e82 rpc error: code = Internal desc = unable to Delete Dir using FS [fs1] Relative SymLink [spectrum-scale-csi-volume-store/.volumes/pvc-261406a1-923b-4cc0-bad4-bebf037acf94]. Error [unable to delete dir spectrum-scale-csi-volume-store/.volumes/pvc-261406a1-923b-4cc0-bad4-bebf037acf94:[EFSSG0264C The path /ibm/fs1/spectrum-scale-csi-volume-store/.volumes/pvc-261406a1-923b-4cc0-bad4-bebf037acf94 does not exist.]]



### Data Collection and Debugging

CSI Snap: `/scale-csi/D.1173/csi_snap.tar.gz`
badri-pathak commented 1 week ago

@shrutinipane If the fixes has been verified , can we close this issue?

Jainbrt commented 1 week ago

yes we should close this.