search

web-platform-tests / interop

web-platform-tests Interop project
273 stars 28 forks source link

Monitoring for tests with precondition failures, harness errors, or other problems with the tests #659

Open foolip opened 2 months ago

foolip commented 2 months ago

In https://github.com/web-platform-tests/interop/issues/319 there was a problem with "PRECONDITION_FAILED" in some WebCodecs tests.

I think the conclusion is that we can't have tests like these in Interop, because there's no good way to score them.

We should have a system for finding such tests and periodically review it. It could be as simple as a (very long) wpt.fyi query.

foolip commented 2 months ago

Something like https://wpt.fyi/results/?label=master&label=experimental&aligned&view=interop&q=label%3Ainterop-2024-indexeddb%20status%3Aerror is a small subset of such a query, currently reveling a harness error in Chrome.