search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
56 stars 93 forks source link

[disrupt_decommission_streaming_err] nemises failed during clean_group0_garbage for target node #8930

Open timtimb0t opened 1 day ago

timtimb0t commented 1 day ago

Packages

Scylla version: 2024.2.0~rc3-20241004.89f8638e9e9b with build-id 95fdcb05a2d968f52fbcfc46114ef612aaf56608

Kernel Version: 5.15.0-1070-aws

Issue description

Describe your issue in detail and steps it took to produce it.

Issue happens after the target node been terminated while clean_group0_garbage invoking (no any response from scylla received)

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 12 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

OS / Image: ami-0ea8ad7f9305a3297 ami-0686dda0ba768267e (aws: undefined_region)

Test: longevity-multidc-schema-topology-changes-12h-test Test id: d96e7c16-1105-4f5d-92bd-2b149705343f Test name: enterprise-2024.2/tier1/longevity-multidc-schema-topology-changes-12h-test Test method: longevity_test.LongevityTest.test_custom_time Test config file(s):

Logs and commands - Restore Monitor Stack command: `$ hydra investigate show-monitor d96e7c16-1105-4f5d-92bd-2b149705343f` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=d96e7c16-1105-4f5d-92bd-2b149705343f) - Show all stored logs command: `$ hydra investigate show-logs d96e7c16-1105-4f5d-92bd-2b149705343f` ## Logs: - **parallel-topology-schema-changes-mu-db-node-d96e7c16-8** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-8-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-8-d96e7c16.tar.gz) - **parallel-topology-schema-changes-mu-db-node-d96e7c16-9** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-9-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-9-d96e7c16.tar.gz) - **parallel-topology-schema-changes-mu-db-node-d96e7c16-11** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-11-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-11-d96e7c16.tar.gz) - **parallel-topology-schema-changes-mu-db-node-d96e7c16-16** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-16-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-16-d96e7c16.tar.gz) - **parallel-topology-schema-changes-mu-db-node-d96e7c16-3** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-3-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-3-d96e7c16.tar.gz) - **parallel-topology-schema-changes-mu-db-node-d96e7c16-4** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-4-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-4-d96e7c16.tar.gz) - **parallel-topology-schema-changes-mu-db-node-d96e7c16-15** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-15-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241006_131751/parallel-topology-schema-changes-mu-db-node-d96e7c16-15-d96e7c16.tar.gz) - **db-cluster-d96e7c16.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/db-cluster-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/db-cluster-d96e7c16.tar.gz) - **sct-runner-events-d96e7c16.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/sct-runner-events-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/sct-runner-events-d96e7c16.tar.gz) - **sct-d96e7c16.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/sct-d96e7c16.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/sct-d96e7c16.log.tar.gz) - **loader-set-d96e7c16.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/loader-set-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/loader-set-d96e7c16.tar.gz) - **monitor-set-d96e7c16.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/monitor-set-d96e7c16.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d96e7c16-1105-4f5d-92bd-2b149705343f/20241007_000638/monitor-set-d96e7c16.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/enterprise-2024.2/job/tier1/job/longevity-multidc-schema-topology-changes-12h-test/7/) [Argus](https://argus.scylladb.com/test/5cd4bfe3-231b-4f48-99bc-f21ebd1b5efe/runs?additionalRuns[]=d96e7c16-1105-4f5d-92bd-2b149705343f)
soyacz commented 1 day ago

@aleksbykov can you take a look?