Closed dimakr closed 2 weeks ago
hmm, shouldn't nemesis select a node that is not the target_node in parallel nemesis?
the problem is that they both are the first
sdcm.nemesis.SisyphusMonkey: Current Target: Node parallel-topology-schema-changes-mu-db-node-97c11d18-7 [13.40.68.247 | 10.3.1.199] (dc name: eu-west-2scylla_node_west, rack: 2a) with running nemesis: None
running nemesis: None
is the problem, it means both select a node, while that node isn't decorated with a running nemesis yet
Packages
Scylla version:
2024.1.8-20240724.fc3e399a25f3
with build-id646cf933d8926947ade5b2a7cbc5bacb145df4fb
Kernel Version:5.15.0-1066-aws
Issue description
During enterprise-2024.1/longevity/longevity-multidc-schema-topology-changes-12h-test#26 test
disrupt_terminate_and_replace_node
anddisrupt_create_index
nemeses were started in parallel and were targeted onto the same node-7. NodeTerminateAndReplace nemesis started node termination at 02:43:52 and finished at 02:49:38:CreateIndex tried to start index creation on node-7 at 02:54:47:
and eventually failed as the node was no longer available, with the error:
Impact
Parallel nemeses affected one another in a disruptive manner.
How frequently does it reproduce?
No other occurrences of the issue were noticed.
Installation details
Cluster size: 12 nodes (i3en.2xlarge)
Scylla Nodes used in this run: No resources left at the end of the run
OS / Image:
ami-072fc07743bf86cd3 ami-09a43832bb62c9b19
(aws: undefined_region)Test:
longevity-multidc-schema-topology-changes-12h-test
Test id:97c11d18-65ec-4dfa-9b9d-70ba669c3f11
Test name:enterprise-2024.1/longevity/longevity-multidc-schema-topology-changes-12h-test
Test config file(s):Logs and commands
- Restore Monitor Stack command: `$ hydra investigate show-monitor 97c11d18-65ec-4dfa-9b9d-70ba669c3f11` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=97c11d18-65ec-4dfa-9b9d-70ba669c3f11) - Show all stored logs command: `$ hydra investigate show-logs 97c11d18-65ec-4dfa-9b9d-70ba669c3f11` ## Logs: - **db-cluster-97c11d18.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/db-cluster-97c11d18.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/db-cluster-97c11d18.tar.gz) - **sct-runner-events-97c11d18.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/sct-runner-events-97c11d18.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/sct-runner-events-97c11d18.tar.gz) - **sct-97c11d18.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/sct-97c11d18.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/sct-97c11d18.log.tar.gz) - **loader-set-97c11d18.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/loader-set-97c11d18.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/loader-set-97c11d18.tar.gz) - **monitor-set-97c11d18.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/monitor-set-97c11d18.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/97c11d18-65ec-4dfa-9b9d-70ba669c3f11/20240725_151613/monitor-set-97c11d18.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/enterprise-2024.1/job/longevity/job/longevity-multidc-schema-topology-changes-12h-test/26/) [Argus](https://argus.scylladb.com/test/8764d79e-d785-4b99-9c70-99e2f24d0a18/runs?additionalRuns[]=97c11d18-65ec-4dfa-9b9d-70ba669c3f11)