Web interface for tracking Test Failures

[def-]

Feature request

Currently when a test fails it's difficult to figure out if this is a known issue, since when it's been occurring, how frequently it happens, etc. Some of our buildkite insights tooling is already helpful here, but I think we should have a dashboard where all of that is tracked and easily visible for QA and devs. Buildkite Analytics might seem similar, but would fall short of many of our requirements.

I'm imagining this a bit like the Sentry interface, where each test failure is logged, they are connected based on error message, the github issue is linked.

[def-]

@nrainer-materialize is already starting on uploading ci annotations to the database.

[nrainer-materialize]

I think it would be great if we could split this down into smaller tasks.

[def-]

At the moment this is pretty exploratory for me, so I'm not sure yet what will work or not and what smaller tasks to generate.

Edit: I see these main tasks for now:

1. Implement UI using React (can mock data with json for now, I'll start working on that) Professional mockup:

Scanned Document.pdf

2. Implement a small server that gets us the data we need from Mz (I'll work on that next after part 1 makes sense) Requires these operations: fetch test failures (with offset and limit), fetch issues (with offset and limit)
3. Store the data in Materialize (@nrainer-materialize is working on that) Requires a table for test failures, a table for issues (with regex), and an MV with that issues match their respective test failures

[nrainer-materialize]

Thanks for the additional details. Do you plan to make the first version read-only or will the webserver of step two also conduct writes?

[def-]

So far read-only. I'm not sure if we want to move away from having ci-regexp in github.