search

IBM / ibm-spectrum-scale-csi

The IBM Spectrum Scale Container Storage Interface (CSI) project enables container orchestrators, such as Kubernetes and OpenShift, to manage the life-cycle of persistent storage.
Apache License 2.0
66 stars 49 forks source link

PVC should expand if we increase the values from invalid to valid value #1227

Open saurabhwani5 opened 1 week ago

saurabhwani5 commented 1 week ago

Describe the bug

When we try to expand 1GB PVC size to beyond scale limit for example 9223372036854775807Gi and then we try to increase to 2 GB then in this condition PVC size does not increase and give error as The PersistentVolumeClaim "scale-advance-pvc-1" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

How to Reproduce?

Please list the steps to help development teams reproduce the behavior

  1. Install CSI with following images :

    Operator : quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:1e579401a52f4ae12eff2b0b61b83af12d37d354a80d2ee6b83b9ac6b7cc01e2
Driver : quay.io/badri_pathak/ibm-spectrum-scale-csi-driver:pvc_max_size_v1

  2. Create 1 Gi PVC :

    
[root@ocppr1226]# cat pvc.yaml
apiVersion: v1
kind: Pod
metadata:
name: csi-scale-fsetdemo-pod-2
labels:
app: nginx
spec:
containers:
- name: web-server
 image: nginx:1.22.0
 volumeMounts:
   - name: mypvc
     mountPath: /usr/share/nginx/html/scale
 ports:
 - containerPort: 80
volumes:
- name: mypvc
 persistentVolumeClaim:
   claimName: scale-advance-pvc-1
   readOnly: false

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scale-advance-pvc-1 spec: accessModes:

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ibm-spectrum-scale-csi-advance provisioner: spectrumscale.csi.ibm.com parameters: volBackendFs: "fs0" version: "2" reclaimPolicy: Delete allowVolumeExpansion: true [root@ocppr1226]# oc apply -f pvc.yaml pod/csi-scale-fsetdemo-pod-2 created persistentvolumeclaim/scale-advance-pvc-1 created storageclass.storage.k8s.io/ibm-spectrum-scale-csi-advance created [root@ocppr1226]# [root@ocppr1226]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE scale-advance-pvc-1 Bound pvc-4ed38513-a27a-4693-9625-53cc8442f95e 1Gi RWX ibm-spectrum-scale-csi-advance 59s [root@ocppr1226]#


4. Expand PVC size to 9223372036854775807Gi :

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scale-advance-pvc-1 spec: accessModes:

  1. Expand PVC size to 2 Gi 
    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scale-advance-pvc-1
spec:
accessModes:
- ReadWriteMany
resources:
requests:
  storage: 2Gi
storageClassName: ibm-spectrum-scale-csi-advance
[root@ocppr1226]# oc apply -f pvc.yaml
pod/csi-scale-fsetdemo-pod-2 configured
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-advance unchanged
The PersistentVolumeClaim "scale-advance-pvc-1" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

Expected behavior

PVC size should increase to 2GB as current PVC size is 1GB

Data Collection and Debugging

[root@ocp D.1227]# pwd
/scale-csi/D.1227
[root@ocp D.1227]# ls
mustgather.tar.gz
badri-pathak commented 1 week ago

@saurabhwani5 There are several open issue in the external-resizer regarding this same issue. I haven't found any resolution in the latest external-resizer as such.Still discussion has been added in the https://github.com/kubernetes-csi/external-resizer

We need to wait to get this fixed. Reference : resize failed and can't recover due to pvc rejection "Forbidden: field can not be less than previous value "

There is an upcoming feature to handle such scenario as a part of feature-gate in 1.31 https://kubernetes.io/docs/concepts/storage/persistent-volumes/#recovering-from-failure-when-expanding-volumes

https://github.com/kubernetes-csi/external-resizer?tab=readme-ov-file image

badri-pathak commented 1 week ago

As we have discussed, this is going to be released later in kubernetes releases as this is feature-gates in 1.31 . Keeping this open for now. @saurabhwani5