search

IBM / ibm-spectrum-scale-csi

The IBM Spectrum Scale Container Storage Interface (CSI) project enables container orchestrators, such as Kubernetes and OpenShift, to manage the life-cycle of persistent storage.
Apache License 2.0
66 stars 49 forks source link

CSO status become false in upgrade from 2.10.1 to 2.11.0 #1137

Open saurabhwani5 opened 5 months ago

saurabhwani5 commented 5 months ago

Describe the bug

When tried to upgrade from 2.10.1 to 2.11.0 following error is coming Synchronization of node/driver ibm-spectrum-scale-csi DaemonSet failed for the CSISCaleOperator instance ibm-spectrum-scale-csi CSO status :

[root@csi-kube129-1-x-master ~]# oc get cso -w
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.1    True
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    False
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    True

How to Reproduce?

Please list the steps to help development teams reproduce the behavior

  1. Install CSI 2.10.1:

    [root@csi-kube129-1-x-master ~]# oc get cso -w
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.1    True

  2. Create PVC of version 2 :

    [root@csi-kube129-1-x-master ~]# oc get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
ibm-spectrum-scale-pvc-clone-from-pvc   Bound    pvc-e868506e-245a-4160-bda2-5030c867c6b4   1Gi        RWX            ibm-spectrum-scale-csi-advance   <unset>                 24m
ibm-spectrum-scale-pvc-from-snapshot    Bound    pvc-a405e6a9-7eae-4907-874e-75b5b3f6b8f2   1Gi        RWX            ibm-spectrum-scale-csi-advance   <unset>                 25m
scale-advance-pvc-1                     Bound    pvc-be292e11-0017-460f-9671-7eda17a60bab   1Gi        RWX            ibm-spectrum-scale-csi-advance   <unset>                 33m

  3. Upgrade to 2.11.0 :

    
[root@csi-kube129-1-x-master saurabh]# oc apply -f ibm-spectrum-scale-csi-operator.yaml
deployment.apps/ibm-spectrum-scale-csi-operator configured
clusterrole.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator configured
clusterrolebinding.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator unchanged
serviceaccount/ibm-spectrum-scale-csi-operator unchanged
customresourcedefinition.apiextensions.k8s.io/csiscaleoperators.csi.ibm.com configured
4. Check cso status

[root@csi-kube129-1-x-master ~]# oc get cso -w NAME VERSION SUCCESS ibm-spectrum-scale-csi 2.10.1 True ibm-spectrum-scale-csi 2.11.0 True ibm-spectrum-scale-csi 2.11.0 True ibm-spectrum-scale-csi 2.11.0 False ibm-spectrum-scale-csi 2.11.0 True ibm-spectrum-scale-csi 2.11.0 True


5. Check CSO description:

[root@csi-kube129-1-x-master ~]# oc describe cso Name: ibm-spectrum-scale-csi Namespace: ibm-spectrum-scale-csi-driver Labels: app.kubernetes.io/instance=ibm-spectrum-scale-csi-operator app.kubernetes.io/managed-by=ibm-spectrum-scale-csi-operator app.kubernetes.io/name=ibm-spectrum-scale-csi-operator release=ibm-spectrum-scale-csi-operator Annotations: API Version: csi.ibm.com/v1 Kind: CSIScaleOperator Metadata: Creation Timestamp: 2024-04-24T01:42:35Z Finalizers: finalizer.csiscaleoperators.csi.ibm.com Generation: 2 Resource Version: 6221078 UID: 11932dfe-4d32-4b73-814f-a7f80d31268a Spec: Attacher Node Selector: Key: scale Value: true Clusters: Id: 453210684566980716 Primary: Primary Fs: fs1 Rest API: Gui Host: 10.11.107.129 Secrets: guisecret Secure Ssl Mode: false Id: 4181049054023231843 Rest API: Gui Host: 10.11.57.54 Secrets: remotenewsecret Secure Ssl Mode: false Consistency Group Prefix: 4500ce03-4b53-4a3d-b884-ba5f69868521 Plugin Node Selector: Key: scale Value: true Provisioner Node Selector: Key: scale Value: true Resizer Node Selector: Key: scale Value: true Snapshotter Node Selector: Key: scale Value: true Status: Conditions: Last Transition Time: 2024-04-24T02:21:17Z Message: The CSI driver resources have been created/updated successfully Reason: CSIConfigured Status: True Type: Success Versions: Name: ibm-spectrum-scale-csi Version: 2.11.0 Events: Type Reason Age From Message

Normal CSIConfigured 37m (x4 over 38m) CSIScaleOperator The CSI driver resources have been created/updated successfully Warning UpdateFailed 29s CSIScaleOperator Synchronization of ibm-spectrum-scale-csi-resizer Deployment failed for the CSISCaleOperator instance ibm-spectrum-scale-csi Normal CSIConfigured 15s (x4 over 33s) CSIScaleOperator The CSI driver resources have been created/updated successfully

## Expected behavior
CSO should not give any error log 

## Logs

/scale-csi/D.1137 csisnap.tar.gz

deeghuge commented 3 days ago

@saurabhwani5 do keep eye on this during current upgrade tests