Document the data collection and results generation system

[jgraham]

The way that we go from tests to results to data on wpt.fyi to interop scores is under documented an unclear. There are two specific problems here (which maybe require different solutions, but it's worth considering as part of one higher level issue):

• People don't know what to expect e.g. in terms of latency, and don't know at what point to raise an alarm when something isn't working the way they expect (e.g. when browser changes aren't showing up on the interop dashboard).
• It's unclear what parts of the pipeline are considered stable and can be relied on by external tooling (e.g. for vendors wanting to understand how specific in-progress changes affect interop).

That suggests that for 2024 we should consider two improvements:

• Produce documentation of the overall dataflow, so that there's a reference to point people to for all the systems involved, what they should expect in terms of latency, and what to do when something doesn't work in the way they expect.
• Provide a stable API for accessing the data and tooling needed to recreate the interop scores. Currently there are a variety of data files in different, undocumented, formats, that one must download from different systems, which are quite specialised to the needs of wpt.fyi. That makes external tooling both difficult to create in the first place, and liable to break whenever the needs of wpt.fyi change.

[bkardell]

We agree that these are good/important things to put on our collective priorities list. That said, other than offering to help editorially with documentation or in promoting/discussing documentation when it arrives, there probably isn't a lot that @meyerweb and I can offer in terms of driving those efforts.

[foolip]

I have sent https://github.com/web-platform-tests/interop/pull/667. Editorial review is the main thing that's needed.