While running some performance tests, I noticed that there seems to be a bottleneck that is significantly delaying volumes creation. Here are the results when trying to create 5 volumes per second (see the test scenario at the end of this comment):
After some investigation, I noticed that the bottleneck was the Kubernetes client throttling requests.
What this PR does?:
This PR allows the user to set the QPS and Burst value for the K8S client. Running the same performance test as above with a qps of 50 and a burst of 100 leads to the following result:
Does this PR require any upgrade changes?:
No, if no --kube-api-qps or --kube-api-burst are provided, then the behavior is exactly the same as today.
If the changes in this PR are manually verified, list down the scenarios covered::
Here is ClusterLoader2 scenario used for the performance test presented above:
## Default expected number of PVC is 300
{{$PVCS := DefaultParam .PVCS 300}}
## Set a default timeout of 30 minutes
{{$TIMEOUT := DefaultParam .TIMEOUT "1800s"}}
name: openebs-stress
namespace:
number: 1
tuningSets:
- name: UniformQPS
qpsLoad:
qps: 5
steps:
- name: Create PVCs
phases:
- namespaceRange:
min: 1
max: 1
replicasPerNamespace: {{$PVCS}}
tuningSet: UniformQPS
objectBundle:
- basename: volume-test
objectTemplatePath: "pvc.yaml"
- name: Waiting for PVs to be bound
measurements:
- Identifier: WaitForPVCsToBeBound
Method: WaitForBoundPVCs
Params:
desiredPVCCount: {{$PVCS}}
apiVersion: v1
labelSelector: group = volume-test
timeout: {{$TIMEOUT}}
Why is this PR required? What issue does it fix?:
While running some performance tests, I noticed that there seems to be a bottleneck that is significantly delaying volumes creation. Here are the results when trying to create 5 volumes per second (see the test scenario at the end of this comment):
After some investigation, I noticed that the bottleneck was the Kubernetes client throttling requests.
What this PR does?:
This PR allows the user to set the QPS and Burst value for the K8S client. Running the same performance test as above with a
qps
of50
and aburst
of100
leads to the following result:Does this PR require any upgrade changes?:
No, if no
--kube-api-qps
or--kube-api-burst
are provided, then the behavior is exactly the same as today.If the changes in this PR are manually verified, list down the scenarios covered::
Here is ClusterLoader2 scenario used for the performance test presented above:
With
pvc.yaml
:Checklist:
<type>(<scope>): <subject>