Running WebAssembly Spec Tests in WPT

ajklein commented 11 months ago

Description

As part of the standardization of new WebAssembly features in the W3C's WebAssembly Community Group, spec tests are required as part of the feature advancement process. And passing those tests is required before a feature becomes fully standardized. But WebAssembly spec tests do not come with a testrunner designed to work in a browser. This is essential to their nature, since they're also meant to be able to be run by non-browser implementations, including the reference interpreter.

This proposal is to investigate adding additional test infrastructure to WPT and wpt.fyi to facilitate running WebAssembly spec tests within the WPT framework.

Rationale

The major motivation for taking on this work is to enable WebAssembly features to be submitted as Interop proposals without any additional per-proposal overhead (such as manually duplicating tests from the spec repo into WPT).

A secondary motivation is to improve the transparency of WebAssembly feature interop. The state of the art right now is the Roadmap page on webassembly.org, which is manually curated. Running Wasm spec tests under WPT would make this data available and automatically up-to-date on wpt.fyi.

Investigation Roadmap

The WPT test runner would need to be updated with glue code to adapt Wasm spec tests to the WPT runner
wpt.fyi would need logic to fetch spec tests from https://github.com/WebAssembly/spec/tree/main/test/core (for the currently-standardized spec) or from individual proposals under the WebAssembly Github organization, then run them using the updated runner

foolip commented 11 months ago

I have done a quick experiment with the help of @ahaas. It turns out there's already a build script to wrap Wasm tests in testharness.js, and copying them into WPT (https://github.com/web-platform-tests/wpt/pull/42394) seems to work: https://wpt.fyi/results/wasm/core?sha=3e432312c3

So, the work here is figuring out how to best build and run those tests alongside WPT. A few options on the spectrum from static to dynamic:

Check everything in to WPT and create a CI job to sync the tests
A script in WPT which builds the tests, to be run before invoking ./wpt run (probably with the WebAssembly/spec commit pinned in WPT)
Integration into wptrunner, so that something like ./wpt run --external-suites=wasm will do everything necessary. One could even go so far as to generate the HTML wrappers in wptserve, similar to *.any.js tests.

zcorpan commented 7 months ago

Thank you for proposing Running WebAssembly Spec Tests in WPT for inclusion in Interop 2024.

We are pleased to let you know that this proposal was accepted as part of the WebAssembly Testing investigation. You can follow the progress of this focus area on the Interop 2024 dashboard.

For an overview of our process, see the proposal selection. Thank you for contributing to Interop 2024!

Posted on behalf of the Interop team.

web-platform-tests / interop