search

IBM / ibm-spectrum-scale-csi

The IBM Spectrum Scale Container Storage Interface (CSI) project enables container orchestrators, such as Kubernetes and OpenShift, to manage the life-cycle of persistent storage.
Apache License 2.0
66 stars 49 forks source link

PVCs are not getting bound state with error : specified junction path does not exist #1041

Closed saurabhwani5 closed 1 year ago

saurabhwani5 commented 1 year ago

Describe the bug

When PVC is trying to create with following steps PVC is not getting bound with error: spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-6cbb75b74-sm62g_2a75c8ee-cc4e-4743-9c4c-4fcbac9ad61e failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset": rpc error: code = Internal desc = unable to create directory [pvc-feb001d3-c649-4b38-9116-45c0e0af0d0c/pvc-feb001d3-c649-4b38-9116-45c0e0af0d0c-data] in filesystem [fs0]. Error : [EFSSG0763C The parent directory "/var/gpfs/fs0/pvc-feb001d3-c649-4b38-9116-45c0e0af0d0c/pvc-feb001d3-c649-4b38-9116-45c0e0af0d0c-data" of the specified junction path does not exist.]

How to Reproduce?

  1. Install CSI 2.10.0 with following images:

    [OCP ]# oc get pods -n ibm-spectrum-scale-csi
NAME                                                  READY   STATUS    RESTARTS   AGE
ibm-spectrum-scale-csi-attacher-5bd7bc8569-ft9cb      1/1     Running   0          5h20m
ibm-spectrum-scale-csi-attacher-5bd7bc8569-vkf4r      1/1     Running   0          5h20m
ibm-spectrum-scale-csi-h6tfk                          3/3     Running   0          5h20m
ibm-spectrum-scale-csi-operator-8568d74b64-4k4ft      1/1     Running   0          5h22m
ibm-spectrum-scale-csi-provisioner-6cbb75b74-sm62g    1/1     Running   0          5h20m
ibm-spectrum-scale-csi-resizer-75467db98f-28vwp       1/1     Running   0          5h20m
ibm-spectrum-scale-csi-snapshotter-5f77874594-lhvzm   1/1     Running   0          5h20m
ibm-spectrum-scale-csi-sszk8                          3/3     Running   0          5h20m
ibm-spectrum-scale-csi-vzrnv                          3/3     Running   0          5h20m
[OCP ]# oc get cso -n ibm-spectrum-scale-csi
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.0    True
[OCP ]# oc describe pod -n ibm-spectrum-scale-csi | grep quay
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-220923
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:05837457d22e79593aa476f61374e640b9762e3e53b57b433d08140de70be96d
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator:v2.10.0-220923
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:52a24a97a9ad649603c8708d874c538fc22ef067767761fef9ffae091846695e
  CSI_DRIVER_IMAGE:          quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-220923
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-220923
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:05837457d22e79593aa476f61374e640b9762e3e53b57b433d08140de70be96d
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.10.0-220923
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:05837457d22e79593aa476f61374e640b9762e3e53b57b433d08140de70be96d

  2. Create SC as following :

    [OCP ]# cat sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibm-spectrum-scale-csi-fileset
provisioner: spectrumscale.csi.ibm.com
parameters:
volBackendFs: "fs0"
inodeLimit: "1024"
reclaimPolicy: Delete
[OCP ]# oc apply -f sc.yaml
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-fileset created

  3. Create PVC as following :

    [OCP ]# cat pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scale-fset-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
  storage: 1Gi
storageClassName: ibm-spectrum-scale-csi-fileset
[OCP ]# oc get pvc -w
NAME             STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
scale-fset-pvc   Pending                                      ibm-spectrum-scale-csi-fileset   3s
[OCP ]# oc describe pvc scale-fset-pvc
Name:          scale-fset-pvc
Namespace:     default
StorageClass:  ibm-spectrum-scale-csi-fileset
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
           volume.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
Type     Reason                Age                From                                                                                                               Message
----     ------                ----               ----                                                                                                               -------
Warning  ProvisioningFailed    14s                spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-6cbb75b74-sm62g_2a75c8ee-cc4e-4743-9c4c-4fcbac9ad61e  failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset": rpc error: code = Internal desc = unable to create directory [pvc-feb001d3-c649-4b38-9116-45c0e0af0d0c/pvc-feb001d3-c649-4b38-9116-45c0e0af0d0c-data] in filesystem [fs0]. Error : [EFSSG0763C The parent directory "/var/gpfs/fs0/pvc-feb001d3-c649-4b38-9116-45c0e0af0d0c/pvc-feb001d3-c649-4b38-9116-45c0e0af0d0c-data" of the specified junction path does not exist.]
Normal   Provisioning          13s (x2 over 48s)  spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-6cbb75b74-sm62g_2a75c8ee-cc4e-4743-9c4c-4fcbac9ad61e  External provisioner is provisioning volume for claim "default/scale-fset-pvc"
Normal   ExternalProvisioning  6s (x5 over 48s)   persistentvolume-controller                                                                                        waiting for a volume to be created, either by external provisioner "spectrumscale.csi.ibm.com" or manually created by system administrator
[OCP ]#

    Observation:

    I see it is happening when I have added inode paramter in sc if I remove that and try to create it works

Expected behavior

PVC should bound

Logs:

/scale-csi/D.1041 mustgather.tar.gz

amdabhad commented 1 year ago

As of now this issue is seen on a BVT cluster and it's working fine on other 2 clusters.

Jainbrt commented 1 year ago

@saurabhwani5 I hope we can close this issue, as fix is coming from core filesystem team

saurabhwani5 commented 1 year ago

Right