ceph / ceph-csi

CSI driver for Ceph
Apache License 2.0
1.28k stars 546 forks source link

Add performance test suite to the e2e #2673

Closed humblec closed 2 years ago

humblec commented 2 years ago

Describe the feature you'd like to have

Considering there are possibilities of regression introduction in performance side of the driver, it is good to have some bench marking for our E2E.. There are few options available as mentioned below which has been in use. One of the widely used operator is RIPSAW

Ripsaw (https://github.com/cloud-bulldozer/benchmark-operator) is a benchmark operator for OpenShift and Kubernetes that is used to establish a performance‌ ‌baseline‌ ‌of‌ the ‌cluster‌ ‌by‌ ‌deploying‌ ‌common‌ ‌workloads‌ ‌such‌ ‌as:‌ ‌uperf,‌ ‌iperf3,‌ ‌fio,‌ sysbench, YCSB, pgbench, smallfile, fs-drift, and hammerdb. We can also use our own workload if we want to compared to the workloads built in.

Kubestone (https://operatorhub.io/operator/kubestone) is another Open Source benchmarking Operator that can evaluate the performance of Kubernetes & OpenShift installations. Available on OperatorHub too. It supports variety of benchmarks, such as: CPU & Mem: Sysbench Disk: Fio & IOPing Network: IPerf3 & QPerf Application: Drill & PGbench

We also have other benchmarking suites available like https://github.com/gluster/gluster-performance-test-suite

This issue track the request for adding performance tests here in this repo...

nixpanic commented 2 years ago

I think Ripsaw is commonly used for storage performance testing, and other Red Hat colleagues use that regularly.

How do you want to include this in the e2e framework, though? I would recommend to run a job like this on weekly basis, against a multi-node deployment. You can create a job and pipeline for that in the ci/centos branch.

humblec commented 2 years ago

Cc @mykaul for his thoughts as he requested this in first place :)

humblec commented 2 years ago

I think Ripsaw is commonly used for storage performance testing, and other Red Hat colleagues use that regularly.

Yeah.. it looks like out of the options, this is better .

How do you want to include this in the e2e framework, though? I would recommend to run a job like this on weekly basis, against a multi-node deployment. You can create a job and pipeline for that in the ci/centos branch.

I do agree on above thought @nixpanic , may be nightly is an overhead, considering the load we have on our CI and also while comparing the waittime of e2e tests . Multi node deployment does make more sense for per benchmarks Not sure we will get ( or have the) the resource required for running perf tests. If we can pull that, I think multinode depoyment + weekly should be fine too :+1:

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 2 years ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.