Failed `nodetool cleanup` nemesis is considered as passed

vponomaryov commented 2 years ago

Prerequisites

[x] Are you rebased to master ?
[x] Is it reproducible ?
[x] Did you perform a cursory search if this issue isn't opened ?

Versions

SCT: master
scylla: 4.5.3

Logs

test_id: bdf64109-f68f-4d69-90af-7b49ca9fd5cc
job log: https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/valerii/job/longevity-scylla-operator-large-collections-12h/12

Description

The problem appears in conjunction with this Scylla bug: https://github.com/scylladb/scylla/issues/9917 In this bug report we can read that nodetool cleanup command failed on the last - 4th node, but SCT reported it as passed. The problem here is that loader returned exit code 0 having lots of errors in the output message, which are correct messages. So, need to make SCT understand the result of the loader by message in addition to the exit code, not only exit code.

Also, that nemesis, being last one, is not considered in the test run report at all. So, this run must have had 6 nemesis reported, but there are only 5 previous ones.

Steps to Reproduce

Run large-collections type of a load for Scylla with the nodetool cleanup nemesis.
See that last node never gets nodetool cleanup operation as finished
Look at the test report and see that it doesn't have such nemesis considered at all.

Expected behavior: nodetool cleanup nemesis is considered in the test report as failed

Actual behavior: it is absent and the test run is marked as passed, which is false positive.

github-actions[bot] commented 10 months ago

This issue is stale because it has been open 2 years with no activity. Remove stale label or comment or this will be closed in 2 days.

fruch commented 10 months ago

@vponomaryov

Have we seen it in any other place which isn't an operator based case ?

vponomaryov commented 10 months ago

@vponomaryov

Have we seen it in any other place which isn't an operator based case ?

I haven't.

scylladb / scylla-cluster-tests