New WPT runner design

[domenic]

Edit: most of this was taken care of in #1966. What remains is summarized in https://github.com/tmpvar/jsdom/issues/1958#issuecomment-333435727.

Here are the problems I want to solve:

• Our manifest format is a JavaScript array with comments. It's not easy to read, modify, or write
• We have to list a lot of files individually. That's kind of dumb, when we mostly want to run a whole directory.
• When new WPT get added to a directory for which we believe we implement, we don't notice and enable them. They might pass; they might reveal bugs, or spec changes; we just don't really know. We only notice if a WPT roll uncovers changed or deleted files
• If we start passing a previously-failing test, we don't notice, and thus we lose the regression protection.
• We cannot partially pass a test. If a stricter assertion gets added to a test which we previously passed, we now have to disable it.

Here's the plan:

• A new manifest format. It's directory-based, and you list the failures and timeouts. I think it's going to be to-run-2.yaml from https://gist.github.com/domenic/abaeb197600dad644d3ea8ee8a3f21a4
  • to-run-3.yaml doesn't work, I believe, because we need to distinguish between timeouts and failures
  • Thoughts welcome on whether to-run.yaml might be better
  • I'm not sure what to do with timeouts; running them every time to verify they still time out kind of sucks. For now I'm thinking don't run them, but maybe have a special command that does that we can manually try.
  • This is derived from WPT's manifest tooling which allows you to get a list of all test URLs (including .any.js-derived ones), excluding support files
• An option in the manifest format to list a list of failing assertions in the test, instead of just saying "fail". I'm less sure how this would look. We'd need it to be easily input, e.g. by outputting the failure text from the test runner for easy copying and pasting.

The first part of this is mostly working on my local machine, actually. The biggest work is just converting all the lines from our big old JS array into appropriate directory-scoped YAML documents, especially since some new tests have been added to our directories along the way.

Would love any feedback the contributors have!

[Zirro]

This looks like a great improvement over the current system. While it's just a small detail among many good things, as someone who's had to balance quotation marks or insert missing commas one time too many I'm glad you went with YAML for the manifest ;-)

I am currently looking for a WPT-testing solution for css-object-model that I can keep within the repo rather than having to run it through the separate jsdom repository, without necessarily having to duplicate the whole setup.

As part of this overhaul, perhaps it would make sense to move the infrastructure for the runner to a separate package, which jsdom and its various dependencies can require and run their own list of tests through?

I only recently found that you've put together something along these lines before in wpt-runner, though it's not been updated for a while. Could this replace that, perhaps?

[domenic]

Good thinking. It might not be so easy to pull out in the first pass, especially for jsdom itself where the setup code isn't just shimming certain globals. Similarly I'm not sure how you'd envision it working with CSSOM if you wanted to test e.g. <link rel="stylesheet"> parsing. More detail would be helpful.

[Zirro]

I figured that the test runner could allow supplying an instance of jsdom - this repo would use require("../../") with jsdom already located here, while css-object-model could require jsdom as a dependency. Of course, I'd still have to npm link that if I make local changes to jsdom.

[domenic]

Here's what's left after #1966:

• Run these tests in the browser!
• Run failing tests and check that they still fail. Fail the test run if the expectation is incorrect (i.e if we unexpectedly pass).
• Add the ability to partially pass a test. Where should we store the information? Chromium uses -expected.txt files next to the tests themselves, but they don't use a submodule. Do we create a parallel hierarchy? Do we embed all the information in the yaml file?
• Have some way of running the timeout tests and check that they still timeout, maybe? But don't do this by default since it will take a long time?
• Some way of tracking passing tests so that if tests get moved around we know about it? Is this a real concern?

[TimothyGu]

Run these tests in the browser!

WPT runner currently uses a custom resource loader. It would probably be easiest if we add some kind of parsing hooks for modifying the URL to fetch.

Run failing tests and check that they still fail. Fail the test run if the expectation is incorrect (i.e if we unexpectedly pass).

:+1:

Add the ability to partially pass a test. Where should we store the information?

The main YAML file sounds fine to me, though I'd argue just knowing when a previously failing test started passing is a more important priority.

Have some way of running the timeout tests and check that they still timeout, maybe? But don't do this by default since it will take a long time?

We can do a cron job or something and run it once a week. Travis CI supports that.