search

web-platform-tests / interop

web-platform-tests Interop project
https://wpt.fyi/interop
311 stars 28 forks source link

Requirements for scoring results from Test262 #117

Open foolip opened 2 years ago

foolip commented 2 years ago

I promised in https://github.com/web-platform-tests/interop/issues/110 that I would explain what we'd need to score results from Test262 as part of Interop 2023, so that @bkardell can investigate.

My knowledge of Test262 is very cursory, but I think that only engines (V8, SpiderMoney, JSC, etc.) are tested and not those engines as integrated into browsers (Chrome, Firefox, Safari, etc.). I don't know what the results format is, but https://test262.report/ shows results and it's probably possible to work backwards from there to find sample report files.

To score results from Tests262, we'd ideally want test results from the same browser releases that the rest of tests (in WPT) are run again. And we'd want the results format to be compatible with WPT's wptreport.json format, with either a test or test/subtest structure that can be scored in the same fashion.

The most straightforward way of achieving this might be some kind of adapter to run all of Test262 as part of ./wpt run and get results in the same format. I think that @Ms2ger has been involved in discussions around something like that a few years ago.

foolip commented 2 years ago

I found the prior work/discussions that I vaguely remembered:

@annevk and @jgraham were somewhat involved with that.

foolip commented 2 years ago

Having looked at https://github.com/web-platform-tests/wpt/pull/8980, I think a path forward here could be:

Ms2ger commented 2 years ago

Hmm, would we check in the results of the script somewhere? That seems suboptimal. I also wonder if we could use .any.js to reduce the amount of work done in the script.

foolip commented 2 years ago

Given how many tests there are I'd also prefer to not check them in. That also avoids license questions. It would make more sense to me as either a build step, or a feature of wptrunner to generate the right responses on the fly. Some change to the manifest is also needed.

foolip commented 2 years ago

I've tried reviving the previous attempt in https://github.com/web-platform-tests/wpt/pull/35621. It doesn't work yet, tests fail because assert isn't defined.

foolip commented 1 year ago

Interop 2023 launched yesterday, and we ended up not needing any integration with Test262.

jgraham commented 1 year ago

Note that this isn't quite true, we ended up with a one-off port of the backreference tests. If we want to accept features that are only tested in test262 going forward then we should figure out a better plan for integrating the suites.

gsnedders commented 1 year ago

Let's keep this open as something we should probably try to solve over the following year (or at least decide whether or not to solve!), so we're not hastily trying to do this in the run up to Interop 2024.

foolip commented 8 months ago

We didn't have any proposals in Interop 2024 that needed Test262, and we didn't start an investigation effort for it. We did however start an investigation effort for WebAssembly testing, which might pave the way for something here.