search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.91k stars 3.07k forks source link

`Max restarts exceeded` with Safari if `Develop -> Allow Remote Automation` is disabled #34417

Open saschanaz opened 2 years ago

saschanaz commented 2 years ago

Hard to see the problem as https://web-platform-tests.org/running-tests/safari.html doesn't have it and the message gives no reason why the restart happened. The docs should have it at least, and I wonder the error message can be better.

zcorpan commented 1 year ago

Adding --log-mach - --log-mach-level info as suggested by @gsnedders the output contains (with level WARNING):

webdriver.error.SessionNotCreatedException: session not created (500): Could not create a session: You must enable the 'Allow Remote Automation' option in Safari's Develop menu to control Safari via WebDriver.

Output without extra logging: ``` % wpt run --channel preview safari dom/historical.html /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( Running 1 tests in web-platform-tests /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( CRITICAL Max restarts exceeded Ran 0 tests finished in 4.4 seconds. • 0 ran as expected. 0 tests skipped. CRITICAL No tests ran ``` Output with extra logging: ``` % wpt run --channel preview --log-mach - --log-mach-level info safari dom/historical.html /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( 0:00.01 INFO Set tests_root to /Users/simonpieters/git/web-platform-tests/wpt 0:00.01 INFO Set metadata_root to /Users/simonpieters/git/web-platform-tests/wpt 0:00.01 INFO Set manifest_update to True 0:00.01 INFO Set manifest_download to True 0:02.63 wptserve INFO Starting http server on http://web-platform.test:8000 0:02.75 wptserve INFO Starting http server on http://web-platform.test:8001 0:02.88 wptserve INFO Starting http server on http://web-platform.test:8002 0:03.01 wptserve INFO Starting http server on http://web-platform.test:8003 0:03.14 wptserve INFO Starting https server on https://web-platform.test:8443 0:03.27 wptserve INFO Starting https server on https://web-platform.test:8444 0:03.40 wptserve INFO Starting https server on https://web-platform.test:8445 0:03.53 wptserve INFO Starting https server on https://web-platform.test:8446 0:03.66 wptserve INFO Create socket on: (, , 6, '', ('127.0.0.1', 8888)) 0:03.66 wptserve INFO Bind on: (, , 6, '', ('127.0.0.1', 8888)) 0:03.66 wptserve INFO Listen on: (, , 6, '', ('127.0.0.1', 8888)) 0:03.79 wptserve INFO Create socket on: (, , 6, '', ('127.0.0.1', 8889)) 0:03.79 wptserve INFO Bind on: (, , 6, '', ('127.0.0.1', 8889)) 0:03.79 wptserve INFO Listen on: (, , 6, '', ('127.0.0.1', 8889)) 0:03.92 wptserve INFO Starting http2 server on https://web-platform.test:9000 0:04.42 SUITE_START: web-platform-test - running 1 tests 0:04.42 WARNING Unsupported test type print-reftest for product safari 0:04.42 INFO Using 1 child processes 0:04.94 INFO Starting runner /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( 0:05.14 WARNING Failed to start protocol connection 0:05.14 WARNING Traceback (most recent call last): File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/protocol.py", line 63, in setup self.connect() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 446, in connect self.webdriver.start() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 558, in start value = self.send_command("POST", "session", body=body) File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 622, in send_command raise err webdriver.error.SessionNotCreatedException: session not created (500): Could not create a session: You must enable the 'Allow Remote Automation' option in Safari's Develop menu to control Safari via WebDriver. 0:05.14 INFO Closing logging queue 0:05.14 INFO queue closed 0:05.67 INFO Starting runner /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( 0:05.87 WARNING Failed to start protocol connection 0:05.87 WARNING Traceback (most recent call last): File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/protocol.py", line 63, in setup self.connect() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 446, in connect self.webdriver.start() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 558, in start value = self.send_command("POST", "session", body=body) File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 622, in send_command raise err webdriver.error.SessionNotCreatedException: session not created (500): Could not create a session: You must enable the 'Allow Remote Automation' option in Safari's Develop menu to control Safari via WebDriver. 0:05.87 INFO Closing logging queue 0:05.87 INFO queue closed 0:06.41 INFO Starting runner /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( 0:06.61 WARNING Failed to start protocol connection 0:06.61 WARNING Traceback (most recent call last): File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/protocol.py", line 63, in setup self.connect() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 446, in connect self.webdriver.start() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 558, in start value = self.send_command("POST", "session", body=body) File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 622, in send_command raise err webdriver.error.SessionNotCreatedException: session not created (500): Could not create a session: You must enable the 'Allow Remote Automation' option in Safari's Develop menu to control Safari via WebDriver. 0:06.61 INFO Closing logging queue 0:06.61 INFO queue closed 0:07.14 INFO Starting runner /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( 0:07.34 WARNING Failed to start protocol connection 0:07.34 WARNING Traceback (most recent call last): File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/protocol.py", line 63, in setup self.connect() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 446, in connect self.webdriver.start() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 558, in start value = self.send_command("POST", "session", body=body) File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 622, in send_command raise err webdriver.error.SessionNotCreatedException: session not created (500): Could not create a session: You must enable the 'Allow Remote Automation' option in Safari's Develop menu to control Safari via WebDriver. 0:07.34 INFO Closing logging queue 0:07.34 INFO queue closed 0:07.88 INFO Starting runner /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( 0:08.08 WARNING Failed to start protocol connection 0:08.08 WARNING Traceback (most recent call last): File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/protocol.py", line 63, in setup self.connect() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 446, in connect self.webdriver.start() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 558, in start value = self.send_command("POST", "session", body=body) File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 622, in send_command raise err webdriver.error.SessionNotCreatedException: session not created (500): Could not create a session: You must enable the 'Allow Remote Automation' option in Safari's Develop menu to control Safari via WebDriver. 0:08.08 INFO Closing logging queue 0:08.08 INFO queue closed 0:08.62 INFO Starting runner /Users/simonpieters/git/web-platform-tests/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( 0:08.81 WARNING Failed to start protocol connection 0:08.81 WARNING Traceback (most recent call last): File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/protocol.py", line 63, in setup self.connect() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 446, in connect self.webdriver.start() File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 558, in start value = self.send_command("POST", "session", body=body) File "/Users/simonpieters/git/web-platform-tests/wpt/tools/webdriver/webdriver/client.py", line 622, in send_command raise err webdriver.error.SessionNotCreatedException: session not created (500): Could not create a session: You must enable the 'Allow Remote Automation' option in Safari's Develop menu to control Safari via WebDriver. 0:08.81 INFO Closing logging queue 0:08.81 INFO queue closed 0:08.84 CRITICAL Max restarts exceeded 0:08.84 SUITE_END web-platform-test ~~~~~~~~~~~~~~~~~ Ran 0 checks () Expected results: 0 Unexpected results: 0 Unexpected Results ------------------ CRITICAL Max restarts exceeded 0:08.84 INFO Got 0 unexpected results, with 0 unexpected passes 0:08.92 wptserve INFO Stopped http server on web-platform.test:8000 0:08.92 wptserve INFO Stopped http server on web-platform.test:8001 0:08.93 wptserve INFO Stopped http server on web-platform.test:8003 0:08.93 wptserve INFO Stopped http server on web-platform.test:8002 0:08.93 wptserve INFO Stopped http server on web-platform.test:8445 0:08.93 wptserve INFO Stopped http server on web-platform.test:8444 0:08.93 wptserve INFO Stopped http server on web-platform.test:8446 0:08.93 wptserve INFO Stopped http server on web-platform.test:8443 0:08.93 wptserve INFO Stopped http server on web-platform.test:9000 0:08.96 wptserve INFO Close on: (, , 6, '', ('127.0.0.1', 8889)) 0:08.96 wptserve INFO Close on: (, , 6, '', ('127.0.0.1', 8888)) 0:08.99 INFO Closing logging queue 0:08.99 INFO queue closed 0:08.99 CRITICAL No tests ran ```
gsnedders commented 1 year ago

The grouped logger only outputs ERROR and CRITICAL logs, thus safaridriver giving a useful error message totally gets swallowed by our logging config.

It would probably be good to make some of the startup warnings into errors, and maybe handle webdriver.error.SessionNotCreatedException specifically (because a traceback there isn't very useful in that context, but the message probably is).

That said, it's not totally obvious how to do this without breaking abstraction given executors.protocol is the one that does all the logging when an exception is thrown.