Misleading error message on failing manager test `test_restore_backup_with_task`

[karol-kokoszka]

Prerequisites

• [ ] Are you rebased to master ?
• [ ] Is it reproducible ?
• [ ] Did you perform a cursory search if this issue isn't opened ?

Versions

• SCT: origin/manager-3.2
• scylla: 2022.2

Logs

• test_id: 'TestId': '87caa233-2e37-4a93-8d68-f55a07ad08a9'
• job log: https://jenkins.scylladb.com/view/scylla-manager/job/manager-3.2/job/centos-sanity-test/12/

Description

There is a misleading error output on failures in test_restore_backup_with_task. This test is performing the restore and waits until the restore is done. The timeout is set to 1000 seconds. If the timeout is exceeded, then the whole suit should be aborted and no next tests should be scheduled.

Right now, even tough the assertion, checking if the restore task completed, timed out, another test-case was started. The other test case failed, because it couldn't schedule the repair. It couldn't schedule the repair, because there was already ongoing repair on the node. The ongoing repair was triggered by the failed test_restore_backup_with_task.

Steps to Reproduce

Run https://jenkins.scylladb.com/view/scylla-manager/job/manager-3.2/job/centos-sanity-test.

More detials: https://jenkins.scylladb.com/view/scylla-manager/job/manager-3.2/job/centos-sanity-test/12/consoleFull#2095715862fcc21424-66d2-4bd8-8e0d-9746405e5b16

Expected behavior: Whenever there is a failure on test_restore_backup_with_task it should hard stop and report the error in this test-case.

Actual behavior: test_restore_backup_with_task failed on the task status assertion, but another test-case was started (test_repair_multiple_keyspace_types). The test_repair_multiple_keyspace_types failed ofc, because it wanted to perform the repair, but there was still ongoing repair job started by the test_restore_backup_with_task.

[fgelcer]

i think this is intentional that the test won't abort the whole suite, but perhaps we need to understand how to stop the task that has timed out