search

web-platform-tests / interop

web-platform-tests Interop project
https://wpt.fyi/interop
278 stars 28 forks source link

Wrong WebCodecs score for Safari stable on https://wpt.fyi/interop-2023?stable #618

Closed nt1m closed 8 months ago

nt1m commented 8 months ago

I'm not sure what's going on but https://wpt.fyi/interop-2023?stable shows "87.3%" even though clicking on the focus area link shows 98%: https://wpt.fyi/results/webcodecs?product=chrome&product=firefox&product=safari&aligned=&view=interop&q=label%3Ainterop-2023-webcodecs&label=master&label=stable

I don't really get where this number comes from.

nt1m commented 8 months ago

cc @past @DanielRyanSmith @jgraham Do we know why this is getting computed wrong?

dbaron commented 8 months ago

It looks like it's wrong in the interop view: the 8 tests that end in ?av1 show up as "0 of 1 subtests" in the interop view but are shown as green and it seems like no points are deducted. (Or maybe something is trying to exclude those tests but doing it strangely?)

jgraham commented 8 months ago

The dashboard is handling PRECONDITION_FAILED incorrectly (counting it as passing all the tests).

As previously discussed it doesn't make sense to have tests in Interop that pass in the absence of interoperability, so if Safari isn't going to implement AV1 support these tests should be removed.

nt1m commented 8 months ago

Ah :( This same issue again. I thought we fixed the AV1 tests, but we only did so in STP apparently. I'll file a wpt-metadata PR to get the AV1 tests removed.