search

web-platform-tests / interop

web-platform-tests Interop project
271 stars 28 forks source link

WebCodecs scores inconsistent between test results page and dashboard #319

Open nt1m opened 1 year ago

nt1m commented 1 year ago

https://wpt.fyi/results/webcodecs?label=experimental&label=master&product=chrome&product=firefox&product=safari&aligned&view=interop&q=label%3Ainterop-2023-webcodecs https://wpt.fyi/interop-2023

First link has 72% pass rate, the dashboard shows 59.6%, seems like the scores are computed differently.

gsnedders commented 1 year ago

Could it be that we, once again, have tests labelled that no longer exist?

nt1m commented 1 year ago

@DanielRyanSmith @foolip Do you know what is going on here?

DanielRyanSmith commented 1 year ago

This is a logic error on wpt.fyi. It looks like there are a number of tests being marked as 100% passing, but have subtests with "PRECONDITION FAILED" statuses that are not being aggregated correctly. They're displaying with green backgrounds, indicating passing, but display "0 of X subtests" text in the cell. I'll have to investigate further to determine what is actually happening here.

Screenshot 2023-04-18 at 10 13 47 AM

For now, it is safe to assume that the numbers on the dashboard are correct, and the results page needs a change.

DanielRyanSmith commented 1 year ago

After looking further, this is very likely caused by "PRECONDITION_FAILED" not being explicitly defined as a valid status on the server side of wpt.fyi where search results are aggregated.

The aggregator method that populates search results (e.g. when searching for interop labels) ignores subtests with statuses that it does not recognize.

Note that this does not affect the normal view of wpt.fyi because only search results are aggregated at runtime (most view data is populated using pre-aggregated summary files).

Does anyone have a concrete definition of what "PRECONDITION_FAILED" means as a status? It would be nice to add a definition of what that status means when adding it to the valid status constants.

nt1m commented 1 year ago

@DanielRyanSmith PRECONDITION_FAILED is returned by assert_implements_optional when the thing that is optional is not there. So this should not be counted as a fail. In this case the test results page is correct (and the Interop dashboard is wrong).

See: https://web-platform-tests.org/writing-tests/testharness-api.html#optional-features

nt1m commented 1 year ago

In this case of WebCodecs, it's just hitting this status since the tested video codecs are not supported, and the test does not require them to be supported.

nt1m commented 1 year ago

https://github.com/web-platform-tests/results-analysis/pull/178