search

IBM / ibm-spectrum-scale-csi

The IBM Spectrum Scale Container Storage Interface (CSI) project enables container orchestrators, such as Kubernetes and OpenShift, to manage the life-cycle of persistent storage.
Apache License 2.0
66 stars 49 forks source link

Pod mount fails on power env for CSI 2.10.0 #1063

Closed saurabhwani5 closed 10 months ago

saurabhwani5 commented 11 months ago

Describe the bug

When tried to create PVC and pods on Power 8 env , pod mount fails with error and kubectl exec doesn't work for driver pod: 

 'message': 'MountVolume.SetUp failed for volume '
            '"pvc-2d201d56-471b-44a1-906d-dbf191b967e0" : rpc error: code = '
            'Unknown desc = checkGpfsType: the path '
            '[/ibm/fs1/pvc-2d201d56-471b-44a1-906d-dbf191b967e0/pvc-2d201d56-471b-44a1-906d-dbf191b967e0-data] '
            'is not a valid gpfs path ',
[root@power8-master spectrum-scale-container-automation]# oc exec -it ibm-spectrum-scale-csi-xcdh2 -- bash
Fatal glibc error: CPU lacks ISA 3.00 support (POWER9 or later required)
command terminated with exit code 127

How to Reproduce?

  1. Install CSI on power 8 env having rhel 8.8 : 
    [root@power8-master ~]# oc get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
ibm-spectrum-scale-csi-attacher-759c9d7bd4-86mrd      1/1     Running   0          6s
ibm-spectrum-scale-csi-attacher-759c9d7bd4-rctd6      1/1     Running   0          6s
ibm-spectrum-scale-csi-m4v9t                          3/3     Running   0          6s
ibm-spectrum-scale-csi-operator-764cc4489f-md2r8      1/1     Running   0          51s
ibm-spectrum-scale-csi-provisioner-58b4d75cfd-sbg9h   1/1     Running   0          6s
ibm-spectrum-scale-csi-resizer-6577bb5ddd-zkzxf       1/1     Running   0          6s
ibm-spectrum-scale-csi-snapshotter-85b4fcf68c-fzzc7   1/1     Running   0          6s
ibm-spectrum-scale-csi-sqjcg                          3/3     Running   0          6s
[root@power8-master ~]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.0    True
[root@power8-master ~]# oc describe pod | grep quay
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0
Normal  Pulled     18s   kubelet            Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0" already present on machine
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:e3d2f9fb68b2d7cd1faf84002bb73626da10bed5d81f91945a592d41893e2fda
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:e3d2f9fb68b2d7cd1faf84002bb73626da10bed5d81f91945a592d41893e2fda
  CSI_DRIVER_IMAGE:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0
Normal  Pulling    63s   kubelet            Pulling image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:e3d2f9fb68b2d7cd1faf84002bb73626da10bed5d81f91945a592d41893e2fda"
Normal  Pulled     60s   kubelet            Successfully pulled image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:e3d2f9fb68b2d7cd1faf84002bb73626da10bed5d81f91945a592d41893e2fda" in 3.138822501s (3.138846373s including waiting)
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0
Normal  Pulled     18s   kubelet            Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:57b4ee494ca48342d1ffaf22a166286202b0406b88316e4dcbe87212df6ca8f0" already present on machine
[root@power8-master ~]# hostnamectl
Static hostname: powercluster
     Icon name: computer-vm
       Chassis: vm
    Machine ID: 7cdb87d4aac047cbb999ed09c10c3033
       Boot ID: feaa1fd2bf314e7c9d8604ebba05dda0
Virtualization: kvm
Operating System: Red Hat Enterprise Linux 8.8 (Ootpa)
   CPE OS Name: cpe:/o:redhat:enterprise_linux:8::baseos
        Kernel: Linux 4.18.0-477.21.1.el8_8.ppc64le
  Architecture: ppc64-le
[root@power8-master ~]# lshw
powercluster
description: pSeries Guest
product: linux,kvm Model# JVASB037250B B01
vendor: IBM
serial: JWSBSW16320122
width: 64 bits
capabilities: smp
configuration: uuid=82da1d20-2b29-488b-820c-c8d3463bc645
*-core
   description: Motherboard
   physical id: 0
   capabilities: qemu guest
 *-cpu:0
      description: POWER8 (architected), altivec supported
      product: PowerPC,POWER8
      physical id: 0
      bus info: cpu@0
      version: 2.0 (pvr 004d 0200)
      size: 2893MHz
      configuration: chip-id=0 threads=1
  2. Create PVC and pods as following : 
    
apiVersion: v1
kind: Pod
metadata:
name: csi-scale-fsetdemo-pod-1
labels:
app: nginx
spec:
containers:
- name: web-server
 image: nginx
 volumeMounts:
   - name: mypvc
     mountPath: /usr/share/nginx/html/scale
 ports:
 - containerPort: 80
volumes:
- name: mypvc
 persistentVolumeClaim:
   claimName: scale-advance-pvc-1
   readOnly: false

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scale-advance-pvc-1 spec: accessModes:

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ibm-spectrum-scale-csi-advance provisioner: spectrumscale.csi.ibm.com parameters: volBackendFs: "fs1" version: "2" reclaimPolicy: Delete

3. Check PVC and pod status :

[root@power8-master ~]# oc get pvc -w NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE scale-advance-pvc-1 Pending ibm-spectrum-scale-csi-advance 5s scale-advance-pvc-1 Pending pvc-f2a417f6-34e0-417d-bf86-c967370a35ff 0 ibm-spectrum-scale-csi-advance 56s scale-advance-pvc-1 Bound pvc-f2a417f6-34e0-417d-bf86-c967370a35ff 1Gi RWX ibm-spectrum-scale-csi-advance 56s [root@power8-master ~]# oc get pods NAME READY STATUS RESTARTS AGE csi-scale-fsetdemo-pod-1 0/1 ContainerCreating 0 101s ibm-spectrum-scale-csi-attacher-759c9d7bd4-86mrd 1/1 Running 0 4m56s ibm-spectrum-scale-csi-attacher-759c9d7bd4-rctd6 1/1 Running 0 4m56s ibm-spectrum-scale-csi-m4v9t 3/3 Running 0 4m56s ibm-spectrum-scale-csi-operator-764cc4489f-md2r8 1/1 Running 0 5m41s ibm-spectrum-scale-csi-provisioner-58b4d75cfd-sbg9h 1/1 Running 0 4m56s ibm-spectrum-scale-csi-resizer-6577bb5ddd-zkzxf 1/1 Running 0 4m56s ibm-spectrum-scale-csi-snapshotter-85b4fcf68c-fzzc7 1/1 Running 0 4m56s ibm-spectrum-scale-csi-sqjcg 3/3 Running 0 4m56s

4. Check pod  description:

[root@power8-master ~]# oc describe pod csi-scale-fsetdemo-pod-1 Name: csi-scale-fsetdemo-pod-1 Namespace: ibm-spectrum-scale-csi-driver Priority: 0 Service Account: default Node: power8-worker-1./10.11.23.199 Start Time: Tue, 21 Nov 2023 23:15:32 -0800 Labels: app=nginx Annotations: Status: Pending IP: IPs: Containers: web-server: Container ID: Image: nginx Image ID: Port: 80/TCP Host Port: 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: Mounts: /usr/share/nginx/html/scale from mypvc (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vhpg2 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: mypvc: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: scale-advance-pvc-1 ReadOnly: false kube-api-access-vhpg2: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Warning FailedScheduling 2m46s default-scheduler 0/3 nodes are available: persistentvolumeclaim "scale-advance-pvc-1" not found. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Warning FailedScheduling 2m44s default-scheduler 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Warning FailedScheduling 109s default-scheduler 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Normal Scheduled 105s default-scheduler Successfully assigned ibm-spectrum-scale-csi-driver/csi-scale-fsetdemo-pod-1 to power8-worker-1. Normal SuccessfulAttachVolume 96s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-f2a417f6-34e0-417d-bf86-c967370a35ff" Warning FailedMount 31s (x8 over 95s) kubelet MountVolume.SetUp failed for volume "pvc-f2a417f6-34e0-417d-bf86-c967370a35ff" : rpc error: code = Unknown desc = checkGpfsType: the path [/ibm/fs1/6fdb5da2-1a50-44b0-bfc9-225beaf8f605-ibm-spectrum-scale-csi-driver/pvc-f2a417f6-34e0-417d-bf86-c967370a35ff] is not a valid gpfs path


5. Do kubectl exec for driver pod :

[root@power8-master ~]# oc exec -it ibm-spectrum-scale-csi-sqjcg -- bash Fatal glibc error: CPU lacks ISA 3.00 support (POWER9 or later required) command terminated with exit code 127 [root@power8-master ~]# oc exec -it ibm-spectrum-scale-csi-sqjcg -- sh Fatal glibc error: CPU lacks ISA 3.00 support (POWER9 or later required) command terminated with exit code 127



## Expected behavior
Pods should be attached to PVC  

## Logs :
/scale-csi/D.1063
csisnap.tar.gz
amdabhad commented 11 months ago

Even though we have the fix for one part for power8, CSI driver further fails to mount or unmount the volumes as the library used by CSI driver makes the os calls which fail again with power8 + UBI9 combination.

amdabhad commented 10 months ago

Closing this as UBI9+power8 causes other issues and we have dropped support of power8 from CSI 2.10 onward.