log improvement for shallow copy volumes

saurabhwani5 commented 9 months ago

Describe the bug

For not supported shallow copy volumes creation, we are giving only one error message as CreateVolume ValidateShallowCopyVolume failed, we can add more logs here to get detailed information about the exact issue in shallow copy creation.

How to Reproduce?

Install CSI with shallow copy images #1083 :

[OCP~ ]# oc get pods
NAME                                                  READY   STATUS    RESTARTS      AGE
ibm-spectrum-scale-csi-5cckq                          3/3     Running   0             4d
ibm-spectrum-scale-csi-attacher-585ddbd6bb-4tv8g      1/1     Running   0             4d
ibm-spectrum-scale-csi-attacher-585ddbd6bb-gfzhp      1/1     Running   0             4d
ibm-spectrum-scale-csi-c89q4                          3/3     Running   0             4d
ibm-spectrum-scale-csi-fqcjn                          3/3     Running   0             4d
ibm-spectrum-scale-csi-operator-f775c8ffd-twvqj       1/1     Running   1 (23h ago)   4d
ibm-spectrum-scale-csi-provisioner-765d779f6b-zb5v5   1/1     Running   0             4d
ibm-spectrum-scale-csi-resizer-644c7bd59b-95kjv       1/1     Running   0             4d
ibm-spectrum-scale-csi-snapshotter-7cd97548d5-j82pf   1/1     Running   0             4d
[OCP~ ]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.11.0    True
[OCP~ ]# oc describe pod | grep quay
Image:         quay.io/hemalatha_gajendran/driver_shallowcopy_clone_final:latest
Image ID:      quay.io/hemalatha_gajendran/driver_shallowcopy_clone_final@sha256:da5f831fe5e257732095c37c8b7a2daaab99dd1f60fab4fa916310dcdde0a513
Image:         quay.io/hemalatha_gajendran/driver_shallowcopy_clone_final:latest
Image ID:      quay.io/hemalatha_gajendran/driver_shallowcopy_clone_final@sha256:da5f831fe5e257732095c37c8b7a2daaab99dd1f60fab4fa916310dcdde0a513
Image:         quay.io/hemalatha_gajendran/driver_shallowcopy_clone_final:latest
Image ID:      quay.io/hemalatha_gajendran/driver_shallowcopy_clone_final@sha256:da5f831fe5e257732095c37c8b7a2daaab99dd1f60fab4fa916310dcdde0a513
Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:18264e0c9c112856bc2744f7f971a4b60ecb24de57e46af4d35456dcdf8e3cbf
Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:18264e0c9c112856bc2744f7f971a4b60ecb24de57e46af4d35456dcdf8e3cbf
  CSI_DRIVER_IMAGE:          quay.io/hemalatha_gajendran/driver_shallowcopy_clone_final:latest

Create source PVC of version 2 as following:


[OCP~ ]# cat apply.yaml
apiVersion: v1
kind: Pod
metadata:
name: csi-scale-fsetdemo-pod-2
labels:
app: nginx
spec:
containers:
- name: web-server
 image: nginx
 volumeMounts:
   - name: mypvc
     mountPath: /usr/share/nginx/html/scale
 ports:
 - containerPort: 80
volumes:
- name: mypvc
 persistentVolumeClaim:
   claimName: scale-advance-pvc-1
   readOnly: false

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scale-advance-pvc-1 spec: accessModes:

ReadWriteMany resources: requests: storage: 1Gi storageClassName: ibm-spectrum-scale-csi-advance

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ibm-spectrum-scale-csi-advance provisioner: spectrumscale.csi.ibm.com parameters: volBackendFs: "fs0" version: "2" reclaimPolicy: Delete [OCP~ ]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE scale-advance-pvc-1 Bound pvc-01e37192-9fbd-457d-b526-406733250c1c 1Gi RWX ibm-spectrum-scale-csi-advance 21m


3. Create snapshot of source PVC:

apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: ibm-spectrum-scale-snapshot spec: volumeSnapshotClassName: ibm-spectrum-scale-snapshotclass-advance source: persistentVolumeClaimName: scale-advance-pvc-1

apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: ibm-spectrum-scale-snapshotclass-advance driver: spectrumscale.csi.ibm.com parameters: snapWindow: "30" #Optional : Time in minutes (default=30) deletionPolicy: Delete [OCP~ ]# oc get vs NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE ibm-spectrum-scale-snapshot true scale-advance-pvc-1 1Gi ibm-spectrum-scale-snapshotclass-advance snapcontent-4e87919c-89f0-425a-bbbc-454bf33b94a5 18m 18m


4. Create shallow copy volume of version 1 from above snapshot :

[OCP~ ]# cat snapshot.yaml apiVersion: v1 kind: Pod metadata: name: csi-scale-fsetdemo-pod-snapshot labels: app: nginx spec: containers:

name: web-server image: docker-na-public.artifactory.swg-devops.com/sys-spectrum-scale-team-test-environment-docker-local/nginx:1.22.0 volumeMounts:
- name: mypvc mountPath: /usr/share/nginx/html/scale ports:
  - containerPort: 80 volumes:
name: mypvc persistentVolumeClaim: claimName: ibm-spectrum-scale-pvc-from-snapshot readOnly: false

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ibm-spectrum-scale-pvc-from-snapshot spec: accessModes:
- ReadOnlyMany resources: requests: storage: 1Gi storageClassName: ibm-spectrum-scale-csi-version1-diff dataSource: name: ibm-spectrum-scale-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io [OCP~ ]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ibm-spectrum-scale-pvc-from-snapshot Pending ibm-spectrum-scale-csi-version1-diff 19m

Check the PVC description

[OCP~ ]# oc describe pvc ibm-spectrum-scale-pvc-from-snapshot
Name:          ibm-spectrum-scale-pvc-from-snapshot
Namespace:     ibm-spectrum-scale-csi
StorageClass:  ibm-spectrum-scale-csi-version1-diff
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
           volume.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
DataSource:
APIGroup:  snapshot.storage.k8s.io
Kind:      VolumeSnapshot
Name:      ibm-spectrum-scale-snapshot
Used By:     csi-scale-fsetdemo-pod-snapshot
Events:
Type     Reason                Age                  From                                                                                                                Message
----     ------                ----                 ----                                                                                                                -------
Warning  ProvisioningFailed    20m                  spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-765d779f6b-zb5v5_72e1334d-6b93-4576-a73e-68061f1a2cd8  failed to provision volume with StorageClass "ibm-spectrum-scale-csi-version1-diff": error getting handle for DataSource Type VolumeSnapshot by Name ibm-spectrum-scale-snapshot: error getting snapshot ibm-spectrum-scale-snapshot from api server: volumesnapshots.snapshot.storage.k8s.io "ibm-spectrum-scale-snapshot" not found
Warning  ProvisioningFailed    20m (x5 over 20m)    spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-765d779f6b-zb5v5_72e1334d-6b93-4576-a73e-68061f1a2cd8  failed to provision volume with StorageClass "ibm-spectrum-scale-csi-version1-diff": error getting handle for DataSource Type VolumeSnapshot by Name ibm-spectrum-scale-snapshot: snapshot ibm-spectrum-scale-snapshot is not Ready
Normal   Provisioning          4m4s (x13 over 20m)  spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-765d779f6b-zb5v5_72e1334d-6b93-4576-a73e-68061f1a2cd8  External provisioner is provisioning volume for claim "ibm-spectrum-scale-csi/ibm-spectrum-scale-pvc-from-snapshot"
Warning  ProvisioningFailed    3m49s (x7 over 19m)  spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-765d779f6b-zb5v5_72e1334d-6b93-4576-a73e-68061f1a2cd8  failed to provision volume with StorageClass "ibm-spectrum-scale-csi-version1-diff": rpc error: code = Internal desc = CreateVolume ValidateShallowCopyVolume failed
Normal   ExternalProvisioning  46s (x85 over 20m)   persistentvolume-controller                                                                                         waiting for a volume to be created, either by external provisioner "spectrumscale.csi.ibm.com" or manually created by system administrator

Expected behavior

Log improvement can be done here to know why validation is failed in shallow copy creation ,we have done log improvement in clone operation where we are giving errors such as

cloning of volume across clusters is not supported
cloning of volumes between different version of storageClass is not supported

we can create same type of error logs in shallow copy

saurabhwani5 commented 7 months ago

one log improvement is required in case of volume expansion of shallow copy volume , I get following log error fileset [pvc-618bb79e-8edc-475f-a22e-a2bde8083bde] does not exist in filesystem [fs0]

Events:
  Type     Reason                 Age                     From                                                                                                                Message
  ----     ------                 ----                    ----                                                                                                                -------
  Normal   Provisioning           10m                     spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5476667595-hwlc4_3498e9f8-283d-47db-b8b9-8786f15c2109  External provisioner is provisioning volume for claim "ibm-spectrum-scale-csi/ibm-spectrum-scale-pvc-from-snapshot-2"
  Normal   ExternalProvisioning   10m (x3 over 10m)       persistentvolume-controller                                                                                         waiting for a volume to be created, either by external provisioner "spectrumscale.csi.ibm.com" or manually created by system administrator
  Normal   ProvisioningSucceeded  10m                     spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5476667595-hwlc4_3498e9f8-283d-47db-b8b9-8786f15c2109  Successfully provisioned volume pvc-618bb79e-8edc-475f-a22e-a2bde8083bde
  Warning  ExternalExpanding      9m35s                   volume_expand                                                                                                       Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Normal   Resizing               4m27s (x13 over 9m35s)  external-resizer spectrumscale.csi.ibm.com                                                                          External resizer is resizing volume pvc-618bb79e-8edc-475f-a22e-a2bde8083bde
  Warning  VolumeResizeFailed     4m23s (x13 over 9m26s)  external-resizer spectrumscale.csi.ibm.com                                                                          resize volume "pvc-618bb79e-8edc-475f-a22e-a2bde8083bde" by resizer "spectrumscale.csi.ibm.com" failed: rpc error: code = Internal desc = fileset [pvc-618bb79e-8edc-475f-a22e-a2bde8083bde] does not exist in filesystem [fs0]. Error [<nil>]

saurabhwani5 commented 7 months ago

log improvement of shallow copy expansion is added in #1109 and verified , thanks @hemalathagajendran

IBM / ibm-spectrum-scale-csi