[dartboard] Automate cluster scalability testing - Githubissues

rancher / qa-tasks

List of QA Backlog

1 stars 1 forks source link

[dartboard] Automate cluster scalability testing #953

Open git-ival opened 1 year ago

git-ival commented 1 year ago

Test Scenario:

Setup HA Rancher cluster with a dedicated monitoring node/agent
- Install rancher-monitoring with rancher performance debugging metrics enabled
Provision downstream clusters in batches
- Default 50 clusters per batch
Continue provisioning clusters until reaching the expected limit of downstream clusters

Note: Collect metrics and other relevant artifacts while provisioning during: 1st batch, halfway-point batch, and final batch

Resource/object counts
Grafana + Prometheus metrics
Total # of Clusters provisioned
pprof cpu and heap logs during: 1st batch, halfway-point batch, final batch
Rancher pod logs
Optional:
- etcd response times
- etcd logs
- Database metrics
- Max, Min, Avg, Median for Rancher Cluster metrics (CPU, Mem, Disk utilization, etc.)
- Max, Min, Avg, Median for Cluster provisioning timings

Relies on the following in order to fully automate:

rancher/qa-tasks#955
rancher/qa-tasks#960

git-ival commented 3 weeks ago

Will need to refactor to fit into rancher/dartboard appropriately. May not be worth it since we should be using k6 for the vast majority of our testing now. I'll leave this open for now