Add wptrunner executor / driver for Deno

[lucacasonato]

Hey :wave:

The Deno project has been using WPT for a few months now to test our spec compliance for Web APIs. We just recently created a dashboard to view WPT results: https://wpt.deno.land

In response @foolip suggested on Twitter that it might be possible for us to upload results directly to https://wpt.fyi. This would be fantastic, but this quickly brings up a challenge: https://wpt.fyi expects results in the standard format that ./wpt run outputs. We currently do not use this same output, because we do not use ./wpt run. Instead we use a custom hand-rolled test runner, with a custom output format, and a custom expectations format. It would be fantastic if instead could reuse the existing wpt infrastructure for this, namely wptrunner and wpt run.

This poses some questions:

1. would the addition be welcome, or would it be considered out of scope?
2. if so, is it possible to add wptrunners that can just run unit (testharness) tests, and no ref or crash tests?
3. is it possible to add wptrunners that can only execute one test file per process? (ie no process reuse between test files)
4. is there an integration guide for new wptrunners?

With some guidance I would be more than happy to put in the work to get this done (assuming the addition is welcome).

[foolip]

Hi @lucacasonato, that's a very cool project, I didn't know about it before.

We're already working separately on uploading a wptreport.json produced by other means, but a Deno executor is worth exploring too.

There is some prior art that seems relevant, namely the "jsshell" scope documented in https://web-platform-tests.org/writing-tests/testharness.html and which I believe @Ms2ger had a hand in. I'm not sure if it will be useful though.

To answer your questions:

would the addition be welcome, or would it be considered out of scope?

I would say it depends on how big the impact on the overall code base is. An executor should be fairly isolated, but you never know what we'll run into since the other executors are for browsers. If it looks like it would require non-trivial work to keep an ExecutorDeno working over time, then the WPT maintainers might be reluctant, or want to see a commitment to the project over time before being comfortable with it. But we have accepted similar things that aren't strictly testing end-user web browsers in the past, it's a tradeoff.

if so, is it possible to add wptrunners that can just run unit (testharness) tests, and no ref or crash tests?

I think so, by making the executor only support testharness.js. But can Deno run testharness.js tests that are embedded in an HTML page, or only *.any.js tests and similar?

is it possible to add wptrunners that can only execute one test file per process? (ie no process reuse between test files)

I think so, it sounds similar to ExecutorServo which is different from most other executors that use WebDriver or similar.

is there an integration guide for new wptrunners?

Not quite I think, but https://web-platform-tests.org/tools/wptrunner/README.html has a bunch of documentation. If there's more, @jgraham would know where to find it.

[lucacasonato]

But can Deno run testharness.js tests that are embedded in an HTML page, or only *.any.js tests and similar?

It can run both, but we prefer any.js ones, because those usually don't rely on document.

[foolip]

@lucacasonato so I take it that Deno has an HTML parser and can execute inline scripts and so on, then?

[lucacasonato]

Our WPT runner does, yeah.

[foolip]

Does this mean that parsing HTML is a feature of the runner, and not something Deno can do natively? Does the runner extract the script and hope it'll work without the surrounding HTML, or something like that?

[lucacasonato]

Exactly. Deno has no understanding of HTML. The runner will parse the HTML file, extract all external and internal scripts, and will load then into V8 in the right order and type depending on if it is a module of "regular" script. It will also replace "testharnessreport.js" with our own version of that file that doesn't use the DOM, and instead outputs results to stderr (Servo does the same). Source code for this extraction is at https://github.com/denoland/deno/blob/6ee983b12741d2c38b7d2babb88f5ace317dc4b4/tools/wpt/runner.ts#L141-L176

[foolip]

I see, interesting :)

Obviously this is a way of running the tests that's not officially supported and there's going to be lots of ways to break it, but if it works some of the time, that's something!

And importantly it means that an executor wouldn't need to distinguishing between different kinds of testharness.js tests, which I was afraid it might.

[jgraham]

So, to answer the original questions:

would the addition be welcome, or would it be considered out of scope?

IMO it would be welcome, particuarly if we have a way to do basic smoketesting in CI (e.g. running the infrastructure tests) and someone with an interest in maintainaing it.

if so, is it possible to add wptrunners that can just run unit (testharness) tests, and no ref or crash tests?

Yes.

is it possible to add wptrunners that can only execute one test file per process? (ie no process reuse between test files)

Yes.

is there an integration guide for new wptrunners?

No. But https://web-platform-tests.org/tools/wptrunner/docs/design.html might be helpful.

To expand out a bit on the one-word answers above, each "product" in wptrunner is represented by a module in tools/wptrunner/wptrunner/browsers/. That has a __wptrunner__ dict that specifies configuration for that product. It's not well documented, but you should be able to start by copying one of the existing files. Important properties include the "browser", which is (the name of) the class that owns the browser instance, and the "executor", which is a mapping of test-type to (the name of) a class which actually executes that test type. If you only support testharness.js tests you simply only define that test type as having an implementation.

The usual split in wptrunner between browser and executor is set up around remote automation where we have a browser instance and can attach to it remotely using an executor to run tests of a particular type. But that doesn't apply when we need a process per test. In that case we typically use the NullBrowser class to provide a no-op implementation of the Browser interface, and have the executor do the job of actually starting the binary required to run each test. The Servo implementation works like this, so that's worth having a look at.

The actual process of running the tests is handled by a per-protocol, per-test-type Executor subclass. Since Deno isn't using an existing protocol (like WebDriver) you'll basically end up implementing DenoTestharnessExecutor in tools/wptrunner/wptrunner/executors. If you look in the base class you can hopefully work out what's needed; again the Servo implementation in executorservo.py is likely to be the closest to what you want. Because you're not using a remote protocol like WebDriver, there will be some parts that aren't very relevant (e.g. you can't run testdriver tests, so all the Protocol class stuff is mostly noise to you).

Hopefully that's enough to get you going. If you also want to do upfront test filtering (e.g. to only try to run tests that are likely to work without having to enable/disable individual tests) that's a harder problem and might require some changes at the manifest level which would require an RFC. But if you're happy to just use the metadata files to disable tests you can't run I don't think there's anything that requires broader changes here.