search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
58 stars 95 forks source link

Error taking monitor screenshot: Unable to run "ls /home/scylla-test/sct-monitoring | grep scylla-monitoring-src": failed connecting to "10.142.15.198" #3104

Closed dkropachev closed 1 year ago

dkropachev commented 3 years ago

https://jenkins.scylladb.com/view/scylla-operator/job/operator-master/job/longevity-scylla-operator-3h-gke-grow-shrink/13 Test-id: 087d06f6-dae3-4961-8f96-d2f5023fe10f

Failed to take screenshot:

19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR > Error taking monitor screenshot: Message: timeout: Timed out receiving message from renderer: -3.517
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >   (Session info: chrome=87.0.4280.66)
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR > , traceback: Traceback (most recent call last):
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >   File "/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/sdcm/logcollector.py", line 477, in get_grafana_screenshot
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >     self.remote_browser.get_screenshot(grafana_url, screenshot_path,
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >   File "/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/sdcm/utils/remotewebbrowser.py", line 106, in get_screenshot
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >     self.browser.get(url)
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >   File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >     self.execute(Command.GET, {'url': url})
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >   File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >     self.error_handler.check_response(response)
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >   File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >     raise exception_class(message, screen, stacktrace)
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR > selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: -3.517
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR >   (Session info: chrome=87.0.4280.66)
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:485  c:sdcm.logcollector    p:ERROR > 
19:45:23  < t:2021-01-04 12:45:19,640 f:logcollector.py l:422  c:sdcm.logcollector    p:INFO  > Grafana - browser quit
19:52:27  < t:2021-01-04 12:52:13,487 f:decorators.py   l:66   c:sdcm.utils.decorators p:ERROR > 'get_grafana_snapshot': Number of retries exceeded!
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  > 2021-01-04 12:52:13.488: (TestFrameworkEvent Severity.ERROR), source=LongevityTest.save_email_data() message=save_email_data (silenced) failed with:
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  > exception=Unable to run "ls /home/scylla-test/sct-monitoring | grep scylla-monitoring-src": failed connecting to "10.142.15.198" during 60s
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  > Traceback (most recent call last):
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/./sct.py", line 856, in <module>
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     cli()
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 764, in __call__
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     return self.main(*args, **kwargs)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 717, in main
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     rv = self.invoke(ctx)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in invoke
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     return _process_result(sub_ctx.command.invoke(sub_ctx))
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 956, in invoke
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     return ctx.invoke(self.callback, **ctx.params)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 555, in invoke
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     return callback(*args, **kwargs)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/./sct.py", line 618, in run_test
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     unittest.main(module=None, argv=['python -m unittest', argv],
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/main.py", line 101, in __init__
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     self.runTests()
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/main.py", line 271, in runTests
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     self.result = testRunner.run(self.test)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/runner.py", line 176, in run
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     test(result)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/suite.py", line 84, in __call__
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     return self.run(*args, **kwds)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/suite.py", line 122, in run
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     test(result)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/suite.py", line 84, in __call__
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     return self.run(*args, **kwds)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/suite.py", line 122, in run
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     test(result)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/case.py", line 653, in __call__
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     return self.run(*args, **kwds)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/sdcm/tester.py", line 352, in run
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     result = super(ClusterTester, self).run(self.result)
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/case.py", line 596, in run
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     self._callTearDown()
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/usr/local/lib/python3.9/unittest/case.py", line 553, in _callTearDown
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     self.tearDown()
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/sdcm/tester.py", line 1995, in tearDown
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     self.save_email_data()
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >   File "/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/sdcm/tester.py", line 177, in decor
19:52:27  < t:2021-01-04 12:52:13,497 f:file_logger.py  l:78   c:sdcm.sct_events.file_logger p:INFO  >     self._store_test_result(args[0], exc, exc.__traceback__, name)
19:52:27  < t:2021-01-04 12:52:13,937 f:docker_utils.py l:229  c:sdcm.utils.docker_utils p:INFO  > Container <61ec01b211 longevity-scylla-operator-3h-gke-je-localhost-605635d9-123f-4a04-95db-5c6a9eac10cf-rsyslog> destroyed
19:52:27  < t:2021-01-04 12:52:14,258 f:docker_utils.py l:229  c:sdcm.utils.docker_utils p:INFO  > Container <7464ffb98d longevity-scylla-operator-3h-gke-je-localhost-605635d9-123f-4a04-95db-5c6a9eac10cf-helm> destroyed
19:52:27  < t:2021-01-04 12:52:14,259 f:tester.py       l:2529 c:LongevityTest        p:WARNING > Email is not configured: False or no email data: None
19:52:27  < t:2021-01-04 12:52:26,524 f:setup.py        l:83   c:sdcm.sct_events.setup p:INFO  > Statistics of sent/received events (by device): {
19:52:27  < t:2021-01-04 12:52:26,524 f:setup.py        l:83   c:sdcm.sct_events.setup p:INFO  >     "EventsProcessesRegistry[lod_dir=/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/20210104-080440-909383,id=0x7fd2d8cc1a00,default=True][EVENTS_FILE_LOGGER]": 170,
19:52:27  < t:2021-01-04 12:52:26,524 f:setup.py        l:83   c:sdcm.sct_events.setup p:INFO  >     "EventsProcessesRegistry[lod_dir=/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/20210104-080440-909383,id=0x7fd2d8cc1a00,default=True][EVENTS_GRAFANA_ANNOTATOR]": 170,
19:52:27  < t:2021-01-04 12:52:26,524 f:setup.py        l:83   c:sdcm.sct_events.setup p:INFO  >     "EventsProcessesRegistry[lod_dir=/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/20210104-080440-909383,id=0x7fd2d8cc1a00,default=True][EVENTS_GRAFANA_AGGREGATOR]": 168,
19:52:27  < t:2021-01-04 12:52:26,524 f:setup.py        l:83   c:sdcm.sct_events.setup p:INFO  >     "EventsProcessesRegistry[lod_dir=/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/20210104-080440-909383,id=0x7fd2d8cc1a00,default=True][EVENTS_GRAFANA_POSTMAN]": 167,
19:52:27  < t:2021-01-04 12:52:26,524 f:setup.py        l:83   c:sdcm.sct_events.setup p:INFO  >     "EventsProcessesRegistry[lod_dir=/jenkins/slave/workspace/operator-master/longevity-scylla-operator-3h-gke-add-remove-rack/scylla-cluster-tests/20210104-080440-909383,id=0x7fd2d8cc1a00,default=True][MainDevice]": 170
19:52:27  < t:2021-01-04 12:52:26,524 f:setup.py        l:83   c:sdcm.sct_events.setup p:INFO  > }
github-actions[bot] commented 1 year ago

This issue is stale because it has been open 2 years with no activity. Remove stale label or comment or this will be closed in 2 days.

github-actions[bot] commented 1 year ago

This issue was closed because it has been stalled for 2 days with no activity.