foolip
commented
4 months ago I like the suggestion of a Universally Passing metric. This represents the size of the interoperable web platform, and is something we want to be always growing. In this graph from 2019, it would be the green 4/4 slice:
This graph is test-weighted and that's also what we do for Interop. Still, a all(status:pass) shows the tests and subtests that would contribute to a Universally Passing metric.
I'd also like to have simple pass and fail counts for all browsers (products) in wpt.fyi, and the N/4 (or N/3) style metric from my screenshot above. (BSF is the 2/3 slice in such a metric, and can be broken down further by browser. Similarly, the 1/3 slice could be broken down by browser.)
I guess it is within our charter to deal with the BSF chart, but I assume that that also might include things related to it: For example, making that chart itself more understandable by linking an 'understanding this chart' link and linking to (at least what is queryable) somehow that the chart represents. But also, perhaps the idea that we should have other views. I've been thinking about this a lot and I've written a few posts about it. The last one looks at the idea of tracking/bubbling up data related to "universally passing tests" - see https://bkardell.com/blog/WPT-Dashboard.html
I think this could be interesting and helpful for helping us align and accomplish a lot, as well as portray a kind of 'score' that is similar to BSF, but aimed differently: How well are we working together over time? I think that is a pretty interesting and important metric too.
(related to #270)