search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.98k stars 3.09k forks source link

`wpt serve` should exit with non-zero if any port is already bound #7928

Open lukebjerring opened 6 years ago

lukebjerring commented 6 years ago

This causes all ref-tests to pass (since the test/ref 404 pages are the same!)

Steps to reproduce:

Expected: Fails or errors the test Actual: Still requests on port 8000, and passes any ref tests

RByers commented 6 years ago

Yikes, false positives isn't good. Tentatively marking as priority:roadmap.

alijuma commented 6 years ago

This has priority:roadmap; is there any work on this planned for this quarter?

foolip commented 6 years ago

This came up in triage again today. I tried to repro myself, with ./wpt run --no-manifest-update firefox css/css-color/color-00*.html, which does pass normally. (It hangs after the tests have run, but that's a different matter.) Running ./wpt serve at the same time, which uses port 8000, I see this in the output:

 0:05.59 INFO Starting http server on 127.0.0.1:8001
 0:05.59 INFO Starting http server on 127.0.0.1:8000
 0:05.59 INFO Starting http server on 127.0.0.1:8443

And then "Testing http://web-platform.test:8000/css/css-color/color-001.html == http://web-platform.test:8000/css/css-color/greentext-ref.html", and some stuff in the ./wpt serve log, so yeah, ./wpt run really is using the other thing serving on port 8000 and if it doesn't happen to be wptserve, this won't work at all.

@gsnedders, there's code that tries to make a connection after starting all the servers, but seems like if wptserve fails to start because the port it taken, that's just silently ignored? Or is the problem elsewhere?

gsnedders commented 6 years ago

It's just silently ignored IIRC. I think we get a WARNING in the log, when it really should be a fatal error.

foolip commented 6 years ago

Alright, turning that into a fatal errors makes sense.

foolip commented 6 years ago

Included in Ecosystem Infra 2018 Q4 OKRs

mdittmer commented 5 years ago

Ping from your friendly neighbourhood ecosystem infra rotation

Any updates on this, @gsnedders?

gsnedders commented 5 years ago

Not been worked on.

gsnedders commented 5 years ago

With Chrome we get:

 0:07.08 INFO STDERR: Socket error on port 8000
 0:07.08 WARNING Failed to start HTTP server. You may need to edit /etc/hosts or similar, see README.md.
 0:07.08 INFO Starting http server on web-platform.test:8001
 0:07.09 WARNING Failed to start HTTP server. You may need to edit /etc/hosts or similar, see README.md.
 0:07.09 INFO STDERR: Socket error on port 8443

With Firefox we get:

 0:06.90 INFO Starting http server on 127.0.0.1:8001
 0:06.90 INFO Starting http server on 127.0.0.1:8000
 0:06.90 INFO Starting https server on 127.0.0.1:8443

and it just keeps going (failing in the testharness.js case given we don't have the wptrunner testharnessreport.js loaded).

gsnedders commented 4 years ago

./wpt run now fails:

gsnedders@vanveen:~/Documents/other-projects/wpt/web-platform-tests$ ./wpt run --binary `which chromium` --log-mach-level debug --log-mach - chrome infrastructure/assumptions/ahem.html 
 0:00.01 INFO Using webdriver binary /usr/bin/chromedriver
 0:00.10 vcs DEBUG git rev-parse --show-toplevel
 0:00.10 vcs DEBUG git rev-parse --show-cdup
 0:00.11 vcs DEBUG git rev-parse HEAD
 0:05.30 INFO Using 1 client processes
 0:05.35 INFO STDERR: Socket error on port 8000
 0:05.35 CRITICAL Failed to start HTTP server. Please ensure all the necessary WPT subdomains are mapped to a loopback device in /etc/hosts. See https://github.com/web-platform-tests/wpt#running-the-tests for instructions.
 0:05.35 INFO Starting http server on 127.0.0.1:8001
 0:05.38 INFO Starting https server on 127.0.0.1:8443
 0:10.42 INFO Closing logging queue
 0:10.42 INFO queue closed
Traceback (most recent call last):
  File "./wpt", line 12, in <module>
    wpt.main()
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/wpt/wpt.py", line 165, in main
    rv = script(*args, **kwargs)
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/wpt/run.py", line 796, in run
    rv = run_single(venv, **kwargs) > 0
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/wpt/run.py", line 803, in run_single
    return wptrunner.start(**kwargs)
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/wptrunner.py", line 358, in start
    rv = not run_tests(**kwargs) or logged_critical.has_log
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/wptrunner.py", line 201, in run_tests
    test_environment.ensure_started()
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/environment.py", line 219, in ensure_started
    ", ".join("%s:%s" % item for item in failed))
EnvironmentError: Servers failed to start: http:8000
gsnedders@vanveen:~/Documents/other-projects/wpt/web-platform-tests$ echo $?
1

./wpt serve seems to exit with 0:

CRITICAL:web-platform-tests:Failed to start HTTP server. Please ensure all the necessary WPT subdomains are mapped to a loopback device in /etc/hosts. See https://github.com/web-platform-tests/wpt#running-the-tests for instructions.
Socket error on port 8000
Process http on port 8000:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/serve/serve.py", line 419, in create_daemon
DEBUG:web-platform-tests:Route pattern: ^/tools/runner/update\_manifest\.py$
    self.daemon = init_func(host, port, paths, routes, bind_address, config, **kwargs)
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/serve/serve.py", line 551, in start_http_server
    latency=kwargs.get("latency"))
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/wptserve/wptserve/server.py", line 674, in __init__
    http2=http2)
  File "/mnt/home-extra/gsnedders/Documents/other-projects/wpt/web-platform-tests/tools/wptserve/wptserve/server.py", line 181, in __init__
    BaseHTTPServer.HTTPServer.__init__(self, hostname_port, request_handler_cls, **kwargs)
  File "/usr/lib/python2.7/SocketServer.py", line 420, in __init__
DEBUG:web-platform-tests:Route pattern: ^/tools/runner/(.*)$
    self.server_bind()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 108, in server_bind
    SocketServer.TCPServer.server_bind(self)
  File "/usr/lib/python2.7/SocketServer.py", line 434, in server_bind
    self.socket.bind(self.server_address)
  File "/usr/lib/python2.7/socket.py", line 228, in meth
DEBUG:web-platform-tests:Route pattern: ^/(.*)$
    return getattr(self._sock,name)(*args)
error: [Errno 98] Address already in use
INFO:web-platform-tests:Starting http server on web-platform.test:57297
DEBUG:web-platform-tests:Route pattern: ^/(.*\.py)$
DEBUG:web-platform-tests:Route pattern: ^/(.*\.asis)$
DEBUG:web-platform-tests:Route pattern: ^/(.*\.any\.worker\.js)$
DEBUG:web-platform-tests:Route pattern: ^/(.*\.any\.serviceworker\.html)$
DEBUG:web-platform-tests:Route pattern: ^/(.*\.any\.sharedworker\.html)$
DEBUG:web-platform-tests:Route pattern: ^/(.*\.any\.html)$
DEBUG:web-platform-tests:Route pattern: ^/(.*\.window\.html)$
DEBUG:web-platform-tests:Route pattern: ^/(.*\.worker\.html)$
DEBUG:web-platform-tests:Route pattern: ^/(?P<spec>[^/]+)/tools/(.*)$
DEBUG:web-platform-tests:Route pattern: ^/tools/(.*)$
DEBUG:web-platform-tests:Route pattern: ^/\_certs/(.*)$
DEBUG:web-platform-tests:Route pattern: ^/tools/runner/update\_manifest\.py$
DEBUG:web-platform-tests:Route pattern: ^/tools/runner/(.*)$
INFO:web-platform-tests:Starting https server on web-platform.test:8443
INFO:web-platform-tests:1 subprocess exited:
INFO:web-platform-tests:Status of ws on port 34503:     running
INFO:web-platform-tests:Status of wss on port 41371:    running
INFO:web-platform-tests:Status of http on port 8000:    not running
INFO:web-platform-tests:Status of http on port 57297:   running
INFO:web-platform-tests:Status of https on port 8443:   running
gsnedders@vanveen:~/Documents/other-projects/wpt/web-platform-tests$ echo $?
0