concurrent DataVolume creation timeout

[wywself]

What happened: When I use the following code to create a DV, when the concurrency reaches 100, a timeout error will occur.In addition to retrying to create DV, is there any other way to improve the ability of concurrently created datavolume? Thank you.

newDv, err := kubeVirtClient.CdiClient().CdiV1beta1().DataVolumes(imgParam.Namespace).Create(context.TODO(), dv, metav1.CreateOptions{})

The err is as follows

errors.StatusError=Timeout: request did not complete within requested timeout - context deadline exceeded

What you expected to happen: Supports a large number of concurrent creation of DVs

How to reproduce it (as minimally and precisely as possible): The number of concurrent DV creations reaches 100

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): 1.58.1
• Kubernetes version (use kubectl version): 1.27.6
• OS (e.g. from /etc/os-release): centos
• Kernel (e.g. uname -a): 5.10.0

[akalenyu]

We have some performance coverage outside of this repo and tbh we've not heard of this scenario being an issue. Could you please attach the DV spec? It's possible that for some specific path the server is more overloaded than for others

[wywself]

@akalenyu Here is my dv yaml.

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  annotations:
    cdi.kubevirt.io/cloneType: csi-clone
    cdi.kubevirt.io/storage.usePopulator: "true"
  name: img-vm-xxx
  namespace: vm-xxx
spec:
  pvc:
    accessModes:
    - ReadWriteMany
    resources:
      requests:
        storage: 50Gi
    storageClassName: ceph-block
    volumeMode: Block
  source:
    pvc:
      name: img-h08nsyzj-ceph-block
      namespace: default

If 10 datavolumes are created concurrently, each create takes about 5 seconds. If 50 datavolumes are created concurrently, each create method takes 15 seconds. If 100 datavolumes are created concurrently, a timeout error will be reported or pvc is lost.

newDv, err := kubeVirtClient.CdiClient().CdiV1beta1().DataVolumes(imgParam.Namespace).Create(context.TODO(), dv, metav1.CreateOptions{})