search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
5k stars 3.1k forks source link

testrunner raises exception when no tests are run #48263

Closed gsnedders closed 1 month ago

gsnedders commented 1 month ago

For example:

gsnedders@gsnedders-milk web-platform-tests % ./wpt run --log-mach=- --no-pause --no-fail-on-unexpected --this-chunk=1 --total-chunks=10 --chunk-type=hash --test-types=print-reftest safari
/Volumes/gsnedders/projects/wpt/web-platform-tests/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 3.3.6'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
 0:00.04 INFO Set tests_root to /Volumes/gsnedders/projects/wpt/web-platform-tests
 0:00.04 INFO Set metadata_root to /Volumes/gsnedders/projects/wpt/web-platform-tests
 0:00.04 INFO Set manifest_update to True
 0:00.04 INFO Set manifest_download to True
 0:00.06 INFO Interpreting channel 'nightly' as 'preview'
 0:26.30 wptserve INFO Starting http server on http://web-platform.test:8002
 0:26.30 wptserve INFO Starting http server on http://web-platform.test:8000
 0:26.30 wptserve INFO Starting http server on http://web-platform.test:8001
 0:26.30 wptserve INFO Starting http server on http://web-platform.test:8003
 0:26.30 wptserve INFO Starting https server on https://web-platform.test:8443
 0:26.32 wptserve INFO Starting https server on https://web-platform.test:8445
 0:26.35 wptserve INFO Starting https server on https://web-platform.test:8446
 0:26.38 wptserve INFO Starting https server on https://web-platform.test:8444
 0:26.39 wptserve INFO Create socket on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 0:26.39 wptserve INFO Bind on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 0:26.39 wptserve INFO Listen on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 0:26.39 wptserve INFO Create socket on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 0:26.39 wptserve INFO Bind on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 0:26.39 wptserve INFO Listen on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 0:26.41 wptserve INFO Starting http2 server on https://web-platform.test:9000
 0:26.96 SUITE_START: web-platform-test - running 41 tests
 0:26.96 WARNING Unsupported test type print-reftest for product safari
 0:26.96 INFO Using 1 child processes
 0:26.96 INFO No more tests
 0:26.96 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 0 checks ()
Expected results: 0
Unexpected results: 0
OK
 0:26.96 INFO Got 0 unexpected results, with 0 unexpected passes
 0:26.96 INFO STDERR: Exception in thread 
 0:26.96 INFO STDERR: TestRunnerManager-0
 0:26.96 INFO STDERR: :
 0:26.96 INFO STDERR: Traceback (most recent call last):
 0:26.96 INFO STDERR:   File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
 0:26.96 INFO STDERR:     
 0:26.96 INFO STDERR: self.run()
 0:26.96 INFO STDERR:   File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run
 0:26.96 INFO STDERR:     
 0:26.96 INFO STDERR: self._target(*self._args, **self._kwargs)
 0:26.96 INFO STDERR:   File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 472, in run_loop
 0:26.96 INFO STDERR:     
 0:26.96 INFO STDERR: self._cleanup_run_loop()
 0:26.96 INFO STDERR:   File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 494, in _cleanup_run_loop
 0:26.96 INFO STDERR:     
 0:26.96 INFO STDERR: self.stop_runner(force=force_stop)
 0:26.96 INFO STDERR:   File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 960, in stop_runner
 0:26.97 INFO STDERR:     
 0:26.97 INFO STDERR: self.browser.stop(force=True)
 0:26.97 INFO STDERR: AttributeError
 0:26.97 INFO STDERR: : 
 0:26.97 INFO STDERR: 'NoneType' object has no attribute 'stop'
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8001
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8000
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8002
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8003
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8443
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8444
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8445
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8446
 0:27.46 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 0:27.46 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 0:27.46 wptserve INFO Stopped http server on web-platform.test:9000
 0:27.61 INFO Closing logging queue
 0:27.61 INFO queue closed
 0:27.61 CRITICAL No tests ran

This is almost certainly specific to running an unsupported test type; if you pass a path with no tests, then we simply get a critical error like "Unable to find any tests at the path(s)".

gsnedders commented 1 month ago

You can also hit this with overly many chunks: --this-chunk=1 --total-chunks=100000000 --chunk-type=hash --test-type=testharness hits this too.

jonathan-j-lee commented 1 month ago

That's my bad: #48030 removes the early out in TestRunnerManager.stop_runner() if test_runner_proc is None. I'll forward-fix.

(Unrelated aside: #48238 adds print-reftest support for any browsers that speak WebDriver.)

gsnedders commented 1 month ago

(Unrelated aside: #48238 adds print-reftest support for any browsers that speak WebDriver.)

(It doesn't entirely; e.g. https://github.com/web-platform-tests/wpt/blob/309f3039afc13e006c7c02271c086d11b048f40d/tools/wptrunner/wptrunner/browsers/safari.py#L22-L25 is still required for each browser to specify what executor to use for each test type. And safaridriver doesn't support the underlying WebDriver command.)