search

IBM / ibm-spectrum-scale-csi

The IBM Spectrum Scale Container Storage Interface (CSI) project enables container orchestrators, such as Kubernetes and OpenShift, to manage the life-cycle of persistent storage.
Apache License 2.0
66 stars 49 forks source link

Provisioner and Resizer are going in CrashLoopBackOff State #985

Closed saurabhwani5 closed 1 year ago

saurabhwani5 commented 1 year ago

Describe the bug

When 50k independent PVCs are created after approx 2857 PVC getting in bound state. Provisioner is went to CrashLoopBackOff State due to OOMKilled

How to Reproduce?

  1. Install CSI 2.9.0 : 
    [root@saurabh5-master ~]# oc get pods
NAME                                                  READY   STATUS             RESTARTS          AGE
ibm-spectrum-scale-csi-45jxf                          3/3     Running            0                 22h
ibm-spectrum-scale-csi-attacher-86964567b6-nnwqc      1/1     Running            1 (22h ago)       22h
ibm-spectrum-scale-csi-attacher-86964567b6-wmvmq      1/1     Running            1 (22h ago)       22h
ibm-spectrum-scale-csi-mgml8                          3/3     Running            1 (19h ago)       22h
ibm-spectrum-scale-csi-operator-7dbfbd4d78-t6ddz      1/1     Running            0                 22h
ibm-spectrum-scale-csi-provisioner-666f944b47-8jxsw   0/1     CrashLoopBackOff   136 (2m40s ago)   22h
ibm-spectrum-scale-csi-resizer-68dc77f57b-rvmtj       1/1     Running            160 (3h8m ago)    22h
ibm-spectrum-scale-csi-snapshotter-549b7b4c7d-429q4   1/1     Running            0                 22h
[root@saurabh5-master ~]#
[root@saurabh5-master ~]#
[root@saurabh5-master ~]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.9.0     True
Images 
[root@saurabh5-master 50K]# oc describe pods | grep Image
Image:         quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver@sha256:573b3b2d349359d7871d53060a0fc7df6e03de2e2900d1be46b4146ab1972fb7
Image ID:      quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver@sha256:573b3b2d349359d7871d53060a0fc7df6e03de2e2900d1be46b4146ab1972fb7
Image:         registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4a4cae5118c4404e35d66059346b7fa0835d7e6319ff45ed73f4bba335cf5183
Image ID:      registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4a4cae5118c4404e35d66059346b7fa0835d7e6319ff45ed73f4bba335cf5183
Image:         registry.k8s.io/sig-storage/livenessprobe@sha256:2b10b24dafdc3ba94a03fc94d9df9941ca9d6a9207b927f5dfd21d59fbe05ba0
Image ID:      registry.k8s.io/sig-storage/livenessprobe@sha256:2b10b24dafdc3ba94a03fc94d9df9941ca9d6a9207b927f5dfd21d59fbe05ba0
Image:         registry.k8s.io/sig-storage/csi-attacher@sha256:08721106b949e4f5c7ba34b059e17300d73c8e9495201954edc90eeb3e6d8461
Image ID:      registry.k8s.io/sig-storage/csi-attacher@sha256:08721106b949e4f5c7ba34b059e17300d73c8e9495201954edc90eeb3e6d8461
Image:         registry.k8s.io/sig-storage/csi-attacher@sha256:08721106b949e4f5c7ba34b059e17300d73c8e9495201954edc90eeb3e6d8461
Image ID:      registry.k8s.io/sig-storage/csi-attacher@sha256:08721106b949e4f5c7ba34b059e17300d73c8e9495201954edc90eeb3e6d8461
Image:         quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver@sha256:573b3b2d349359d7871d53060a0fc7df6e03de2e2900d1be46b4146ab1972fb7
Image ID:      quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver@sha256:573b3b2d349359d7871d53060a0fc7df6e03de2e2900d1be46b4146ab1972fb7
Image:         registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4a4cae5118c4404e35d66059346b7fa0835d7e6319ff45ed73f4bba335cf5183
Image ID:      registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4a4cae5118c4404e35d66059346b7fa0835d7e6319ff45ed73f4bba335cf5183
Image:         registry.k8s.io/sig-storage/livenessprobe@sha256:2b10b24dafdc3ba94a03fc94d9df9941ca9d6a9207b927f5dfd21d59fbe05ba0
Image ID:      registry.k8s.io/sig-storage/livenessprobe@sha256:2b10b24dafdc3ba94a03fc94d9df9941ca9d6a9207b927f5dfd21d59fbe05ba0
Image:         quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator@sha256:da7ada19c06b20edc9b3c8067a8380f6879899022dda8a5c1cbed7c15b2a381d
Image ID:      quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator@sha256:36f410d228bd5329444ed82e17ad996103f166aad58db2a17b4fa5ce2601db15
Image:         registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79
Image ID:      registry.k8s.io/sig-storage/csi-provisioner@sha256:7f0d080e35ac455c52956e451b8d230c1eca1a8e0d943fdb657df4e093d03d0f
Image:         registry.k8s.io/sig-storage/csi-resizer@sha256:3a7bdf5d105783d05d0962fa06ca53032b01694556e633f27366201c2881e01d
Image ID:      registry.k8s.io/sig-storage/csi-resizer@sha256:3a7bdf5d105783d05d0962fa06ca53032b01694556e633f27366201c2881e01d
Image:         registry.k8s.io/sig-storage/csi-snapshotter@sha256:0d8d81948af4897bd07b86046424f022f79634ee0315e9f1d4cdb5c1c8d51c90
Image ID:      registry.k8s.io/sig-storage/csi-snapshotter@sha256:0d8d81948af4897bd07b86046424f022f79634ee0315e9f1d4cdb5c1c8d51c90
  2. Create sc and PVC as below 
    
[root@saurabh5-master 50K]# cat sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibm-spectrum-scale-csi-fileset-independent
provisioner: spectrumscale.csi.ibm.com
parameters:
volBackendFs: "fs1"
inodeLimit: "1024"
reclaimPolicy: Delete

Create 50 K PVCs as following: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scale-fset-independent-pvc spec: accessModes:

Expected behavior

All pods should be in running state

saurabhwani5 commented 1 year ago

I have added top command output here to track memory usage : /scale-csi/D.993