ErichDonGubler / moz-webgpu-cts

An extremely fast (but opinionated) tool for working with WPT metadata while developing an implementation of WebGPU in a web browser.
Apache License 2.0
0 stars 2 forks source link

Add `update-backlog` subcommand with first `promote-perma-passing` preset #83

Closed ErichDonGubler closed 1 month ago

ErichDonGubler commented 3 months ago

We at Mozilla recently added the implementation-status: backlog property to all test cases (see upstream docs on metadata), so we can create a workflow for incrementally making WebGPU CTS visible to Firefox's CI sheriffing team. The idea is to incrementally remove it as we determine contradictory test results are worth filing bugs for. We now use implementation-status for both tier 2 and tier 3 of Firefox CI. I'm currently calling the migration of a test from a less stable tier to a more stable tier by removing implementation-status: backlog a "promotion".

We do experiments in promoting tests according to heuristics like "promote permanently PASSing tests" (already implemented here), "promote tests that aren't observed to FAIL or CRASH" (to be implemented in #109), with more to come. The workflow then becomes:

  1. Run tests, and gather wptreport.json files.
  2. Run update-expected as desired for backlogged tests, and create a commit based on changes.
  3. Run update-backlog to tentatively promote tests. Commit and submit to CI as an experiment.
  4. Check tentatively promoted tests are successful in the above experiment. Where they are not, demote them, preferably with bugs in Bugzilla.

We consider using implementation-status: backlog to be valuable because:


Original OP I'm currently planning on separating the corpus of WebGPU CTS test runs in Firefox CI into testing based on the `implementation-status` (see [bug 1873687](https://bugzilla.mozilla.org/show_bug.cgi?id=1873687)). I'm using this PR as a way to explore automatically changing the `implementation-status`.
ErichDonGubler commented 1 month ago

Updated the OP with an edited version of the content I wrote of https://github.com/ErichDonGubler/moz-webgpu-cts/issues/106#issuecomment-2142447507 (CC @sagudev).