We at Mozilla recently added the implementation-status: backlog property to all test cases (see upstream docs on metadata), so we can create a workflow for incrementally making WebGPU CTS visible to Firefox's CI sheriffing team. The idea is to incrementally remove it as we determine contradictory test results are worth filing bugs for. We now useimplementation-status for both tier 2 and tier 3 of Firefox CI. I'm currently calling the migration of a test from a less stable tier to a more stable tier by removing implementation-status: backlog a "promotion".
We do experiments in promoting tests according to heuristics like "promote permanently PASSing tests" (already implemented here), "promote tests that aren't observed to FAIL or CRASH" (to be implemented in #109), with more to come. The workflow then becomes:
Run tests, and gather wptreport.json files.
Run update-expected as desired for backlogged tests, and create a commit based on changes.
Run update-backlog to tentatively promote tests. Commit and submit to CI as an experiment.
Check tentatively promoted tests are successful in the above experiment. Where they are not, demote them, preferably with bugs in Bugzilla.
We consider using implementation-status: backlog to be valuable because:
wptrunner has CLI support for filtering tests by implementation status, so the lift to adjust our CI was light.
It's a clear signal that a test is not yet considered valuable to run as a blocker in CI yet.
It's orthogonal to expected; sometimes, we want to model that a test should pass, but not block CI on it yet; sometimes, we want a test to be expected to FAIL, and block CI when it starts PASSing. Concretely, the former case came up recently with bug 1897131.
Original OP
I'm currently planning on separating the corpus of WebGPU CTS test runs in Firefox CI into testing based on the `implementation-status` (see [bug 1873687](https://bugzilla.mozilla.org/show_bug.cgi?id=1873687)). I'm using this PR as a way to explore automatically changing the `implementation-status`.
We at Mozilla recently added the
implementation-status: backlog
property to all test cases (see upstream docs on metadata), so we can create a workflow for incrementally making WebGPU CTS visible to Firefox's CI sheriffing team. The idea is to incrementally remove it as we determine contradictory test results are worth filing bugs for. We now useimplementation-status
for both tier 2 and tier 3 of Firefox CI. I'm currently calling the migration of a test from a less stable tier to a more stable tier by removingimplementation-status: backlog
a "promotion".We do experiments in promoting tests according to heuristics like "promote permanently
PASS
ing tests" (already implemented here), "promote tests that aren't observed toFAIL
orCRASH
" (to be implemented in #109), with more to come. The workflow then becomes:wptreport.json
files.update-expected
as desired forbacklog
ged tests, and create a commit based on changes.update-backlog
to tentatively promote tests. Commit and submit to CI as an experiment.We consider using
implementation-status: backlog
to be valuable because:wptrunner
has CLI support for filtering tests by implementation status, so the lift to adjust our CI was light.expected
; sometimes, we want to model that a test should pass, but not block CI on it yet; sometimes, we want a test to be expected toFAIL
, and block CI when it startsPASS
ing. Concretely, the former case came up recently with bug 1897131.Original OP
I'm currently planning on separating the corpus of WebGPU CTS test runs in Firefox CI into testing based on the `implementation-status` (see [bug 1873687](https://bugzilla.mozilla.org/show_bug.cgi?id=1873687)). I'm using this PR as a way to explore automatically changing the `implementation-status`.