search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
55 stars 93 forks source link

SCT NodeSetupFailed: 'sudo systemctl disable apt-daily-upgrade.timer' Exit code: 1 #7614

Open temichus opened 3 months ago

temichus commented 3 months ago

SCT failed at node setUp with error 

2024-06-05 15:43:21.222: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=66622741-62a9-4dd7-9792-00c818ed8b72, source=LongevityTest.SetUp()
exception=[<sdcm.cluster_aws.MonitorSetAWS object at 0x7fbdd1820100>]:
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/common.py", line 481, in run
result = future.result(time_out)
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/common.py", line 457, in inner
return_val = fun(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 952, in <lambda>
func=(lambda m: m.wait_for_init()),
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 3887, in wrapper
verify_node_setup_or_startup(start_time, setup_queue, setup_results)
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 3835, in verify_node_setup_or_startup
raise NodeSetupFailed(
sdcm.cluster.NodeSetupFailed: [Node longevity-10gb-3h-6-0-monitor-node-d0363ba1-1 [3.238.80.148 | 10.12.1.45]] NodeSetupFailed: Encountered a bad command exit code!
Command: 'sudo systemctl disable apt-daily-upgrade.timer'
Exit code: 1
Stdout:
Stderr:
Failed to disable unit: Message recipient disconnected from message bus without replying
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 3807, in node_setup
cl_inst.node_setup(_node, **setup_kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 5327, in node_setup
node.disable_daily_triggered_services()
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 2971, in disable_daily_triggered_services
self.remoter.sudo('systemctl disable apt-daily-upgrade.timer')
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/base.py", line 123, in sudo
return self.run(cmd=cmd,
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 614, in run
result = _run()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 70, in inner
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 605, in _run
return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 538, in _run_execute
result = connection.run(**command_kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 620, in run
return self._complete_run(channel, exception, timeout_reached, timeout, result, warn, stdout, stderr)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 655, in _complete_run
raise UnexpectedExit(result)
sdcm.remote.libssh2_client.exceptions.UnexpectedExit: Encountered a bad command exit code!
Command: 'sudo systemctl disable apt-daily-upgrade.timer'
Exit code: 1
Stdout:
Stderr:
Failed to disable unit: Message recipient disconnected from message bus without replying

Installation details

Cluster size: 6 nodes (i4i.2xlarge)

Scylla Nodes used in this run:

OS / Image: ami-05adb73a2c6507653 (aws: undefined_region)

Test: longevity-10gb-3h-test Test id: d0363ba1-8836-42f7-9009-bc38e96fcba5 Test name: scylla-6.0/longevity/longevity-10gb-3h-test Test config file(s):

Logs and commands - Restore Monitor Stack command: `$ hydra investigate show-monitor d0363ba1-8836-42f7-9009-bc38e96fcba5` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=d0363ba1-8836-42f7-9009-bc38e96fcba5) - Show all stored logs command: `$ hydra investigate show-logs d0363ba1-8836-42f7-9009-bc38e96fcba5` ## Logs: - **db-cluster-d0363ba1.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/db-cluster-d0363ba1.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/db-cluster-d0363ba1.tar.gz) - **sct-runner-events-d0363ba1.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/sct-runner-events-d0363ba1.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/sct-runner-events-d0363ba1.tar.gz) - **sct-d0363ba1.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/sct-d0363ba1.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/sct-d0363ba1.log.tar.gz) - **loader-set-d0363ba1.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/loader-set-d0363ba1.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/loader-set-d0363ba1.tar.gz) - **monitor-set-d0363ba1.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/monitor-set-d0363ba1.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/monitor-set-d0363ba1.tar.gz) - **parallel-timelines-report-d0363ba1.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/parallel-timelines-report-d0363ba1.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/d0363ba1-8836-42f7-9009-bc38e96fcba5/20240605_154612/parallel-timelines-report-d0363ba1.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-6.0/job/longevity/job/longevity-10gb-3h-test/6/) [Argus](https://argus.scylladb.com/test/8ec41744-5821-4bc6-b160-0354fb67e3f7/runs?additionalRuns[]=d0363ba1-8836-42f7-9009-bc38e96fcba5)
fruch commented 3 months ago

keeping it for reference, even that it happens very rarely

soyacz commented 3 months ago

we just merged a change in that part, let's see if it helps/breaks.