web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.98k stars 3.09k forks source link

Document output formats from `wpt run` #16288

Open zcorpan opened 5 years ago

zcorpan commented 5 years ago

https://web-platform-tests.org/running-tests/from-local-system.html says how to run tests but doesn't say how to see the results. Currently when running tests the output in the console is:

$ wpt run chrome loading/lazyload/
Running 14 tests in web-platform-tests

Ran 14 tests finished in 9.0 seconds.
  • 14 ran as expected. 0 tests skipped.

$

I want to know how which tests passed/failed. The docs doesn't say how to get this information. It does say this at the end

For details on the supported products and a large number of other options for customising the test run:

./wpt run --help

From there I can learn about output logging options. I wouldn't know which of the logging formats would be reasonably human-readable, suitable to write to stdout, though.

This could be better documented, and maybe the summary by default could say something more than how many tests ran and were skipped.

cc @jugglinmike

jugglinmike commented 5 years ago

Much of this functionality is provided by the mozlog utility. It's actually very tightly integrated with the WPT CLI itself, and this makes documentation a bit more challenging. Not only do changes need to go through that project's review process and release pipeline, they must also be appropriate for non-WPT users of the utility.

About a year ago, feeling the same confusion that @zcorpan reported above, I tried to improve things in the mozlog utility. The full text now includes instructions for where to find more information:

Output Logging:
  Each option represents a possible logging format and takes a filename to
  write that format to, or '-' to write to stdout. Some options are provided
  by the mozlog utility; see https://firefox-source-
  docs.mozilla.org/mozbase/mozlog.html for extended documentation.

  --log-unittest LOG_UNITTEST
                        Unittest style output (provided by mozlog)
  --log-wptreport LOG_WPTREPORT
                        wptreport format
  --log-raw LOG_RAW     Raw structured log messages (provided by mozlog)

(etc.)

The fact that Simon filed this issue indicates that this note isn't sufficient. We're in better shape today than when Simon requested an improvement because as of gh-16851, the documentation site includes the description of all the CLI arguments. That's probably most useful to folks using the site's built-in "search" feature.

Still, I think the main problem is the vagueness of the statement "provided by mozlog". I've submitted a patch which (among other things) changes that to "refer to the mozlog project for further documentation": https://bugzilla.mozilla.org/show_bug.cgi?id=1574253