After changing the error message mechanism, a "spot terminationn" Error is not displayed in the list of errors or warnings

[Orenef11]

Test details

• Test: upgrade_test.UpgradeTest.test_rolling_upgrade
• Build number: 23
• Backend: gce: us-east1
• Test-id: e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf
• Start time: 2021-01-05 08:04:37
• End time: 2021-01-05 08:51:16
• Started by user: timer
• Scylla version: 4.2.2-0.20201213.696ef24226 (-)
• Instance type: i3.2xlarge
• Number of scylladb nodes: 4

Email message

CRITICAL - [2]

• 2021-01-05 08:38:57.901: (CassandraStressEvent Severity.CRITICAL): type=failure node=Node rolling-upgrade-4-3-debian-buster-loader-node-e7bc8deb-0-1 [35.229.101.78 | 10.142.0.151] (seed: False) Stress command completed with bad status -1: Failed to connect over JMX; not collecting these stats
• 2021-01-05 08:38:58.235: (CassandraStressEvent Severity.CRITICAL): type=failure node=Node rolling-upgrade-4-3-debian-buster-loader-node-e7bc8deb-0-1 [35.229.101.78 | 10.142.0.151] (seed: False) Stress command completed with bad status -1: Failed to connect over JMX; not collecting these stats

ERROR - [2]

• 2021-01-05 08:38:59.047: (TestFrameworkEvent Severity.ERROR), source=UpgradeTest.Kill Stress Threads() message=Kill Stress Threads (silenced) failed with: exception=[Errno None] Unable to connect to port 22 on 10.142.0.151 Traceback (most recent call last): File "./sct.py", line 856, in cli() File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke return callback(args, kwargs) File "./sct.py", line 618, in run_test unittest.main(module=None, argv=['python -m unittest', argv], File "/usr/local/lib/python3.8/unittest/main.py", line 101, in init self.runTests() File "/usr/local/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/local/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/local/lib/python3.8/unittest/suite.py", line 84, in call return self.run(*args, *kwds) File "/usr/local/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/local/lib/python3.8/unittest/suite.py", line 84, in call return self.run(args, kwds) File "/usr/local/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/local/lib/python3.8/unittest/case.py", line 736, in call return self.run(*args, *kwds) File "/jenkins/slave/workspace/scylla-4.3/rolling-upgrade/rolling-upgrade-debian10-test/scylla-cluster-tests/sdcm/tester.py", line 309, in run result = super(ClusterTester, self).run(self.result) File "/usr/local/lib/python3.8/unittest/case.py", line 679, in run self._callTearDown() File "/usr/local/lib/python3.8/unittest/case.py", line 636, in _callTearDown self.tearDown() File "/jenkins/slave/workspace/scylla-4.3/rolling-upgrade/rolling-upgrade-debian10-test/scylla-cluster-tests/sdcm/tester.py", line 1918, in tearDown self.stop_resources() File "/jenkins/slave/workspace/scylla-4.3/rolling-upgrade/rolling-upgrade-debian10-test/scylla-cluster-tests/sdcm/tester.py", line 164, in decor funct(args, **kwargs) File "/jenkins/slave/workspace/scylla-4.3/rolling-upgrade/rolling-upgrade-debian10-test/scylla-cluster-tests/sdcm/tester.py", line 1765, in stop_resources self.kill_stress_thread()
• 2021-01-05 08:41:28.214: (TestFrameworkEvent Severity.ERROR), source=UpgradeTest.test_rolling_upgrade (upgrade_test.UpgradeTest)() message=Traceback (most recent call last): File "/jenkins/slave/workspace/scylla-4.3/rolling-upgrade/rolling-upgrade-debian10-test/scylla-cluster-tests/upgrade_test.py", line 475, in test_rolling_upgrade self.verify_stress_thread(complex_cs_thread_pool) File "/jenkins/slave/workspace/scylla-4.3/rolling-upgrade/rolling-upgrade-debian10-test/scylla-cluster-tests/sdcm/tester.py", line 1176, in verify_stress_thread results, errors = cs_thread_pool.verify_results() File "/jenkins/slave/workspace/scylla-4.3/rolling-upgrade/rolling-upgrade-debian10-test/scylla-cluster-tests/sdcm/stress_thread.py", line 288, in verify_results for future in concurrent.futures.as_completed(self.results_futures, timeout=self.timeout): File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 244, in as_completed waiter.event.wait(wait_timeout) File "/usr/local/lib/python3.8/threading.py", line 558, in wait signaled = self._cond.wait(timeout) File "/usr/local/lib/python3.8/threading.py", line 306, in wait gotit = waiter.acquire(True, timeout) File "/jenkins/slave/workspace/scylla-4.3/rolling-upgrade/rolling-upgrade-debian10-test/scylla-cluster-tests/sdcm/tester.py", line 192, in critical_failure_handler raise AssertionError("Critical Error has failed the test") AssertionError: Critical Error has failed the test

The real error is that the spot was terminated, and therefore the stress command failed.

10:39:10  < t:2021-01-05 08:38:58,108 f:cluster_gce.py  l:150  c:sdcm.cluster_gce     p:WARNING > Node rolling-upgrade-4-3-debian-buster-loader-node-e7bc8deb-0-1 [35.229.101.78 | 10.142.0.151] (seed: False): Error during getting spot termination notification Encountered a bad command exit code!

Logs

• Restore Monitor Stack command: $ hydra investigate show-monitor e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf

• Show all stored logs command: $ hydra investigate show-logs e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf

• prometheus - https://cloudius-jenkins-test.s3.amazonaws.com/e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf/prometheus_snapshot_20210105_085106.tar.gz

• grafana - https://cloudius-jenkins-test.s3.amazonaws.com/e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf/20210105_084128/grafana-screenshot-overview-20210105_084129-rolling-upgrade-4-3-debian-buster-monitor-node-e7bc8deb-0-1.png

• grafana - https://cloudius-jenkins-test.s3.amazonaws.com/e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf/20210105_084128/grafana-screenshot-scylla-per-server-metrics-nemesis-20210105_084526-rolling-upgrade-4-3-debian-buster-monitor-node-e7bc8deb-0-1.png

• grafana - https://cloudius-jenkins-test.s3.amazonaws.com/e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf/20210105_085116/grafana-screenshot-overview-20210105_085117-rolling-upgrade-4-3-debian-buster-monitor-node-e7bc8deb-0-1.png

• grafana - https://cloudius-jenkins-test.s3.amazonaws.com/e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf/20210105_085116/grafana-screenshot-scylla-per-server-metrics-nemesis-20210105_085439-rolling-upgrade-4-3-debian-buster-monitor-node-e7bc8deb-0-1.png

• db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf/20210105_090043/db-cluster-e7bc8deb.zip

• monitor-set - https://cloudius-jenkins-test.s3.amazonaws.com/e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf/20210105_090043/monitor-set-e7bc8deb.zip

• sct-runner - https://cloudius-jenkins-test.s3.amazonaws.com/e7bc8deb-1f08-44b7-a789-9c9a5c6c8eaf/20210105_090043/sct-runner-e7bc8deb.zip

[Orenef11]

FTI @roydahan @fruch

[github-actions[bot]]

This issue is stale because it has been open 2 years with no activity. Remove stale label or comment or this will be closed in 2 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 2 days with no activity.