search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.98k stars 3.09k forks source link

Document how to use `./wpt run --verify` locally #15013

Open foolip opened 5 years ago

foolip commented 5 years ago

I'm finding that when running ./wpt run --verify --channel dev --binarywhich google-chrome-unstablechrome /web-locks/signal.tentative.https.any.serviceworker.html locally (for https://github.com/web-platform-tests/wpt/pull/15012) all I get is "::: Test verification FAIL".

I couldn't guess if this meant the test is flaky or that something went wrong internally.

Looking at taskcluster-run.py I found that passing --log-mach - might fix the problem, and it does cause the extra stuff you see in Taskcluster to be printed.

Without that extra output ./wpt run --verify is not very useful. Might also consider changing the defaults.

qiuzhong commented 5 years ago

I can reproduce this issue. The wpt is based on mozlog utility. If the log formatter is not indicated in the command line explicitly, it's None. Maybe that's why all debug output was not printed. When --log-mach - is passed, the logger is set mach(Means human-readable output) and its value is sys.stdout.

Without that extra output ./wpt run --verify is not very useful. Might also consider changing the defaults.

Agreed. However, mozlog utility supports several log formatters (raw, unittest, xunit, html, mach tbpl, ). Should we change the default logger to mach when --verify option is set in the command line?