Grow-shrink cluster perf test is not using multi-az

[soyacz]

while we configure multi az in grow-shrink cluster perf test, nodetool status show we use anyway only one rack.

Installation details

Cluster size: 3 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

• perf-latency-grow-shrink-ubuntu-db-node-b2e47f7a-6 (54.154.216.214 | 10.4.0.214) (shards: 7)
• perf-latency-grow-shrink-ubuntu-db-node-b2e47f7a-5 (3.253.65.77 | 10.4.0.31) (shards: 7)
• perf-latency-grow-shrink-ubuntu-db-node-b2e47f7a-4 (54.247.48.151 | 10.4.1.10) (shards: 7)
• perf-latency-grow-shrink-ubuntu-db-node-b2e47f7a-3 (3.254.164.240 | 10.4.1.169) (shards: 7)
• perf-latency-grow-shrink-ubuntu-db-node-b2e47f7a-2 (54.76.20.11 | 10.4.2.148) (shards: 7)
• perf-latency-grow-shrink-ubuntu-db-node-b2e47f7a-1 (3.253.62.202 | 10.4.2.207) (shards: 7)

OS / Image: ami-044f45ee3df20f616 (aws: undefined_region)

Test: scylla-master-perf-regression-latency-650gb-grow-shrink Test id: b2e47f7a-3ac6-4e8c-817f-dfb1f518850e Test name: scylla-master/scylla-master-perf-regression-latency-650gb-grow-shrink Test config file(s):

• perf-regression-latency-650gb-grow-shrink.yaml

Logs and commands

- Restore Monitor Stack command: `$ hydra investigate show-monitor b2e47f7a-3ac6-4e8c-817f-dfb1f518850e` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=b2e47f7a-3ac6-4e8c-817f-dfb1f518850e) - Show all stored logs command: `$ hydra investigate show-logs b2e47f7a-3ac6-4e8c-817f-dfb1f518850e` ## Logs: - **db-cluster-b2e47f7a.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/b2e47f7a-3ac6-4e8c-817f-dfb1f518850e/20240428_145423/db-cluster-b2e47f7a.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/b2e47f7a-3ac6-4e8c-817f-dfb1f518850e/20240428_145423/db-cluster-b2e47f7a.tar.gz) - **sct-runner-b2e47f7a.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/b2e47f7a-3ac6-4e8c-817f-dfb1f518850e/20240428_145423/sct-runner-b2e47f7a.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/b2e47f7a-3ac6-4e8c-817f-dfb1f518850e/20240428_145423/sct-runner-b2e47f7a.tar.gz) - **monitor-set-b2e47f7a.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/b2e47f7a-3ac6-4e8c-817f-dfb1f518850e/20240428_145423/monitor-set-b2e47f7a.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/b2e47f7a-3ac6-4e8c-817f-dfb1f518850e/20240428_145423/monitor-set-b2e47f7a.tar.gz) - **loader-set-b2e47f7a.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/b2e47f7a-3ac6-4e8c-817f-dfb1f518850e/20240428_145423/loader-set-b2e47f7a.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/b2e47f7a-3ac6-4e8c-817f-dfb1f518850e/20240428_145423/loader-set-b2e47f7a.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-master/job/scylla-master-perf-regression-latency-650gb-grow-shrink/38/) [Argus](https://argus.scylladb.com/test/72a0f93b-6d0b-4f45-8462-e876a768cf54/runs?additionalRuns[]=b2e47f7a-3ac6-4e8c-817f-dfb1f518850e)

[soyacz]

root cause are missing backports to perf-v14:

• https://github.com/scylladb/scylla-cluster-tests/pull/5934
• https://github.com/scylladb/scylla-cluster-tests/pull/6055/

Ideas:

1. test backports on side branch, if ok backport those two to it. Find also bugfixes patches for those - might be a few.
2. test grow-shrink perf test using master SCT branch
3. create branch-perf-v15

[fruch]

root cause are missing backports to perf-v14:

• feature(aws): add suport to multi az #5934
• feature(multi-az): adapt provision step to multi-az #6055

Ideas:

1. test backports on side branch, if ok backport those two to it. Find also bugfixes patches for those - might be a few.
2. test grow-shrink perf test using master SCT branch
3. create branch-perf-v15

I think #3 since we want to do it anyhow, maybe first get the parallel cluster setup in ? anything else you think worth waiting for ?

[soyacz]

I think #3 since we want to do it anyhow, maybe first get the parallel cluster setup in ? anything else you think worth waiting for ?

Just maybe the new throughput test I was working on.

[fruch]

I think #3 since we want to do it anyhow, maybe first get the parallel cluster setup in ? anything else you think worth waiting for ?

Just maybe the new throughput test I was working on.

and we can backport into it the new things (and fixes as needed)

[fruch]

let's raise it in the meeting, if it's important to fix and support multi-az

[fruch]

meanwhile let run on master, and see what's need adapting for example disable KMS

[soyacz]

master run (defaults, oss): https://jenkins.scylladb.com/job/scylla-staging/job/lukasz/job/scylla-master-perf-regression-latency-650gb-grow-shrink-tablets/2/

[soyacz]

failed due: https://github.com/scylladb/scylla-cluster-tests/issues/7432