The IBM Spectrum Scale Container Storage Interface (CSI) project enables container orchestrators, such as Kubernetes and OpenShift, to manage the life-cycle of persistent storage.
Apache License 2.0
66
stars
49
forks
source link
PVCs are not getting restored when there are 100 snapshots #1017
Restored PVCs are not coming in bound state when source is 100 Snapshots.
Source PVC are 100 on which 100 snapshots (1 snapshot per PVC) are taken is of version 1 Independent having data 400+ MB which is written using ioMixer. From these snapshots PVCs are restored which are failing in this case.
[root@worker0 /]# mmxcp config --get-max-value
[I] Current maximum number of parallel copy and sync commands in this cluster: 10
[root@worker0 /]# mmxcp config --set-max-value 100
[I] Successfully set maximum number of parallel copy and sync commands in this cluster: 100
[root@worker0 /]# mmxcp config --get-max-value
[I] Current maximum number of parallel copy and sync commands in this cluster: 100
Warning ProvisioningFailed 24m spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5fc97ff9df-hcbph_4cd251f1-d538-4c35-9a30-d643f80d2ccb failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset-independent": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning ProvisioningFailed 22m (x5 over 24m) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5fc97ff9df-hcbph_4cd251f1-d538-4c35-9a30-d643f80d2ccb failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset-independent": rpc error: code = Aborted desc = volume creation already in process : pvc-f1e2ff6d-d14b-4995-a2e2-e12b7ca9c092
Normal ExternalProvisioning 2m38s (x105 over 27m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "spectrumscale.csi.ibm.com" or manually created by system administrator
Normal Provisioning 84s (x14 over 27m) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5fc97ff9df-hcbph_4cd251f1-d538-4c35-9a30-d643f80d2ccb External provisioner is provisioning volume for claim "default/ibm-spectrum-scale-pvc-from-snapshot-8"
Warning ProvisioningFailed 84s (x8 over 21m) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5fc97ff9df-hcbph_4cd251f1-d538-4c35-9a30-d643f80d2ccb failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset-independent": rpc error: code = Internal desc = snapshot copy job had failed for snapshot: snapshot-7a8afa1c-a0ff-43af-83b0-951c1bca958a
Observation : There are 51/100 are in pending state.
Error Logs:
`[[EFSSG0264C The path /mnt/local-sample/pvc-e507babf-cbb7-4c44-8263-1882108ae71d/.snapshots/snapshot-363933cf-4000-47be-b385-9da40017378a/pvc-e507babf-cbb7-4c44-8263-1882108ae71d-data does not exist.]]`
## Expected behavior
All PVC should be restored properly and CSI should retry in case of failure
Must Gather Uploaded in` scale-csi/D.1017`
Describe the bug
Restored PVCs are not coming in bound state when source is 100 Snapshots. Source PVC are 100 on which 100 snapshots (1 snapshot per PVC) are taken is of version 1 Independent having data 400+ MB which is written using ioMixer. From these snapshots PVCs are restored which are failing in this case.
How to Reproduce?
Install CSI 2.10.0 - (OCP -local snc)
Set mmxcp threads to 100:
Change the provisioner to 100 worker threads:
Set operator replica to 0 before making this change
Create 100 - version 1 Independent PVCs:
Write data using iotools of 400+ MB in all 100 PVCs:
Take snapshot of each PVC:
[root@local-snc Independent]# cat snap/snap.yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: ibm-spectrum-scale-snapshot namespace: default spec: volumeSnapshotClassName: ibm-spectrum-scale-snapshotclass source: persistentVolumeClaimName: scale-fset-independent-pvc-
[root@local-snc ~]# oc get vs | grep snapshot -c
100
Warning ProvisioningFailed 24m spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5fc97ff9df-hcbph_4cd251f1-d538-4c35-9a30-d643f80d2ccb failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset-independent": rpc error: code = DeadlineExceeded desc = context deadline exceeded Warning ProvisioningFailed 22m (x5 over 24m) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5fc97ff9df-hcbph_4cd251f1-d538-4c35-9a30-d643f80d2ccb failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset-independent": rpc error: code = Aborted desc = volume creation already in process : pvc-f1e2ff6d-d14b-4995-a2e2-e12b7ca9c092 Normal ExternalProvisioning 2m38s (x105 over 27m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "spectrumscale.csi.ibm.com" or manually created by system administrator Normal Provisioning 84s (x14 over 27m) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5fc97ff9df-hcbph_4cd251f1-d538-4c35-9a30-d643f80d2ccb External provisioner is provisioning volume for claim "default/ibm-spectrum-scale-pvc-from-snapshot-8" Warning ProvisioningFailed 84s (x8 over 21m) spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-5fc97ff9df-hcbph_4cd251f1-d538-4c35-9a30-d643f80d2ccb failed to provision volume with StorageClass "ibm-spectrum-scale-csi-fileset-independent": rpc error: code = Internal desc = snapshot copy job had failed for snapshot: snapshot-7a8afa1c-a0ff-43af-83b0-951c1bca958a