openebs / lvm-localpv

Dynamically provision Stateful Persistent Node-Local Volumes & Filesystems for Kubernetes that is integrated with a backend LVM2 data storage stack.
Apache License 2.0
235 stars 92 forks source link

fix(localpv): expose Kubernetes client QPS and Burst as flags #229

Closed barkbay closed 1 year ago

barkbay commented 1 year ago

Why is this PR required? What issue does it fix?:

While running some performance tests, I noticed that there seems to be a bottleneck that is significantly delaying volumes creation. Here are the results when trying to create 5 volumes per second (see the test scenario at the end of this comment):

image

After some investigation, I noticed that the bottleneck was the Kubernetes client throttling requests.

What this PR does?:

This PR allows the user to set the QPS and Burst value for the K8S client. Running the same performance test as above with a qps of 50 and a burst of 100 leads to the following result:

image

Does this PR require any upgrade changes?:

No, if no --kube-api-qps or --kube-api-burst are provided, then the behavior is exactly the same as today.

If the changes in this PR are manually verified, list down the scenarios covered::

Here is ClusterLoader2 scenario used for the performance test presented above:

## Default expected number of PVC is 300
{{$PVCS := DefaultParam .PVCS 300}}

## Set a default timeout of 30 minutes
{{$TIMEOUT := DefaultParam .TIMEOUT "1800s"}}

name: openebs-stress

namespace:
  number: 1

tuningSets:
  - name: UniformQPS
    qpsLoad:
      qps: 5

steps:
  - name: Create PVCs
    phases:
      - namespaceRange:
          min: 1
          max: 1
        replicasPerNamespace: {{$PVCS}}
        tuningSet: UniformQPS
        objectBundle:
          - basename: volume-test
            objectTemplatePath: "pvc.yaml"
  - name: Waiting for PVs to be bound
    measurements:
      - Identifier: WaitForPVCsToBeBound
        Method: WaitForBoundPVCs
        Params:
          desiredPVCCount: {{$PVCS}}
          apiVersion: v1
          labelSelector: group = volume-test
          timeout: {{$TIMEOUT}}

With pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{.Name}}
  labels:
    group: volume-test
spec:
  storageClassName: "openebs-immediate" ## Update to reflect you cs
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Checklist:

niladrih commented 1 year ago

Thanks you for your contribution! This change has been taken up as a part of PR #234, closing this PR.