past
commented
1 year ago In the WPT RFCs and infrastructure meeting yesterday I promised to share more details about how the flakiness detection works in Chromium CI. Here is a publicly accessible recent example, courtesy of @WeizhongX.
If you look closely you see that in step 45 we run blink_wpt_tests, which fail. Then in step 50 the system is trying to query the results database for any pre-existing flakiness. And it was able to decide that the failure in blink_web_tests (step 42) is due to a known flaky test, since there is a record of prior failed runs with that test (check the stdout link). It doesn't see any pre-existing flakiness in the failed blink_wpt_tests though. Then in step 57 it retries (perhaps unnecessarily) the shards with the patch a second time. In step 71, it retries without the patch applied. At this point, any previously failed tests will be run 10 times each. Any single failure here will cause the test to be treated as an existing failure, and the patch will be exonerated. It doesn't detect any failures, so the patch is clearly at fault. Finally in step 78 we can see that the patch was exonerated for the blink_web_tests failure since it was pre-existing, but in step 80 it declares the patch responsible for the blink_wpt_tests failures since reruns without the patch were successful.
In case it is useful, the recipe code for trybots can be found here and the recipe documentation is here as well. There are also a few different internal dashboards and alerts that automate the investigation process and provide aggregate data to browser engineers.
Of course all this is not easy to upstream to WPT infra, but hopefully it provides some pointers on what a future comprehensive solution could look like. For now, disabling the stability checks or making them non-blocking is the most efficient path.
foolip
whimboo
jgraham
This RFC follows a proposal from @jgraham in a discussion on Matrix from a few days ago.