amdabhad
commented
12 months ago While we have the setup where close to 50k fileset based PVCs are created, please capture below things:
- Current resources set on all the CSI pods
- Resources in use while creating PVC and pod
- Time taken for each of the following:
mmlsfileset <fs> <existing fileset name>:[root@saurabh5-scalegui ~]# date; mmlsfileset fs1 check8; date Mon Jul 17 23:13:54 PDT 2023
Unable to start tslsfileset on 'fs1' because conflicting program tslsfileset is running. Waiting until it completes or moves to the next phase, which may allow the current command to start. tslsfileset on 'fs1' is finished waiting. Processing continues ... Filesets in file system 'fs1': Name Status Path check8 Linked /ibm/fs1/pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75/check8 Mon Jul 17 23:21:26 PDT 2023
* REST call to GUI for the same mmlsfilest:[root@saurabh5-master ~]# curl --insecure -u 'username:password' -X GET https://saurabh5-scalegui.fyre.ibm.com:443/scalemgmt/v2/filesystems/fs1/filesets/pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75 { "filesets" : [ { "config" : { "comment" : "Fileset created by IBM Container Storage Interface driver", "created" : "2023-06-08 23:42:28,000", "iamMode" : "off", "id" : 2467, "inodeSpace" : 2467, "inodeSpaceMask" : 2096640, "isInodeSpaceOwner" : true, "maxNumInodes" : 1024, "oid" : 5866, "parentId" : 0, "path" : "/ibm/fs1/pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75", "permissionChangeMode" : "chmodAndSetacl", "rootInode" : 1293418499, "snapId" : 0, "status" : "Linked" }, "filesetName" : "pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75", "filesystemName" : "fs1", "usage" : { "allocatedInodes" : 1024, "inodeSpaceFreeInodes" : 976, "inodeSpaceUsedInodes" : 48, "usedBytes" : 0, "usedInodes" : 48 } } ], "status" : { "code" : 200, "message" : "The request finished successfully." } }
* Create a new fileset using mmcrfileset:[root@saurabh5-scalegui ~]# date; mmcrfileset fs1 check8 --inode-space pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75; date Mon Jul 17 22:08:53 PDT 2023 Fileset check8 created with id 55000 root inode 1293418544. Mon Jul 17 22:08:54 PDT 2023
* Link fileset using mmlinkfileset:[root@saurabh5-scalegui ~]# date; mmlinkfileset fs1 check8 -J /ibm/fs1/pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75/check8; date Mon Jul 17 22:12:30 PDT 2023 Fileset check8 linked at /ibm/fs1/pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75/check8 Mon Jul 17 22:12:31 PDT 2023 [root@saurabh5-scalegui ~]#
* Time taken for PVC from create to go to bound state, also the time taken for the same volumeCreate call in driver logs
* REST call to GUI to create a fileset - for this you will get a job id, keep checking the status of job for the completion using the job id: Took 5 Mins 45 Sec[root@saurabh5-master ~]# curl --insecure -u 'udername:password' -X GET https://saurabh5-scalegui.fyre.ibm.com:443/scalemgmt/v2/jobs/1000000270533 { "jobs" : [ { "jobId" : 1000000270533, "status" : "COMPLETED", "submitted" : "2023-07-17 07:51:06,090", "completed" : "2023-07-17 07:56:51,061", "runtime" : 344971, "request" : { "data" : { "filesetName" : "check3", "inodeSpace" : "pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75" }, "type" : "POST", "url" : "/scalemgmt/v2/filesystems/fs1/filesets" }, "result" : { "progress" : [ ], "commands" : [ "mmcrfileset 'fs1' 'check3' --inode-space 'pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75' --allow-permission-change 'chmodAndSetAcl' ", "mmlinkfileset 'fs1' 'check3' -J '/ibm/fs1/pvc-bbaa3ce6-3126-4efa-b91f-ebbefc559a75/check3' " ], "stdout" : [ "EFSSA0194I Waiting for concurrent operation to complete.", "EFSSA0194I Waiting for concurrent operation to complete.", "EFSSA0194I Waiting for concurrent operation to complete.", "EFSSA0194I Waiting for concurrent operation to complete.", "EFSSA0194I Waiting for concurrent operation to complete.", "EFSSA0194I Waiting for concurrent operation to complete.", "EFSSG0070I File set check3 created successfully.", "EFSSG0078I File set check3 successfully linked.\n" ], "stderr" : [ ], "exitCode" : 0 }, "pids" : [ ] } ], "status" : { "code" : 200, "message" : "The request finished successfully." } }
Describe the bug
Currently, I am conducting tests for the creation of 50,000 PVCs. Out of these, 3,000 are independent PVCs, and the remaining 50,000 are dependent PVCs.
How to Reproduce?
[root@saurabh5-master ~]# oc get pvc | grep scale-fset-dependent-sc-2107-pvc-1 scale-fset-dependent-sc-2107-pvc-1 Bound pvc-99206a74-3840-47b0-9055-eb217b982bfb 1Gi RWX ibm-spectrum-scale-csi-fileset-dependent-2107 21m
[root@saurabh5-master 50K]# cat sc.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ibm-spectrum-scale-csi-fileset-independent provisioner: spectrumscale.csi.ibm.com parameters: volBackendFs: "fs1" inodeLimit: "1024" reclaimPolicy: Delete [root@saurabh5-master 50K]# cat pvc.sh
!/bin/bash
for (( i=1 ; i<=$1 ; i++ )) do echo "apiVersion: v1"> test.yaml echo "kind: PersistentVolumeClaim">> test.yaml echo "metadata:">> test.yaml echo " name: scale-fset-independent-pvc-$i">> test.yaml echo "spec:">> test.yaml echo " accessModes:">> test.yaml echo " - ReadWriteMany">> test.yaml echo " resources:">> test.yaml echo " requests:">> test.yaml echo " storage: 1Gi">> test.yaml echo " storageClassName: ibm-spectrum-scale-csi-fileset-independent">> test.yaml kubectl apply -f test.yaml done
[root@saurabh5-master dep]# cat 3000sc.sh
!/bin/bash
read -p "Enter the path to the file: " file_path
if [[ ! -f "$file_path" ]]; then echo "File not found: $file_path" exit 1 fi
Read the file line by line
i=1 while IFS= read -r line; do
Process each line
done < "$file_path"
[root@saurabh5-master dep]# cat apply.sh
!/bin/bash
for ((i=1; i<=3000; i++)) do while true; do pending_count=$(oc get pvc | grep Pending | wc -l)
done for ((j=1; j<=20; j++)) do echo "apiVersion: v1"> test.yaml echo "kind: PersistentVolumeClaim">> test.yaml echo "metadata:">> test.yaml echo " name: scale-fset-dependent-sc-$i-pvc-$j">> test.yaml echo "spec:">> test.yaml echo " accessModes:">> test.yaml echo " - ReadWriteMany">> test.yaml echo " resources:">> test.yaml echo " requests:">> test.yaml echo " storage: 1Gi">> test.yaml echo " storageClassName: ibm-spectrum-scale-csi-fileset-dependent-$i">> test.yaml kubectl apply -f test.yaml done done