Open davepacheco opened 7 months ago
Otherwise we could maybe use some heuristics from cargo nextest output? Or maybe there's already a parseable form of output?
For parseable output, there are two current options:
:+1:
Also a lot of CI failures aren't Rust test failures (e.g., oxidecomputer/omicron#4085, oxidecomputer/omicron#2689, oxidecomputer/omicron#4165) and many of our projects don't use nextest
so I imagine we'll want some kind of heuristic fallback anyway.
I suspect producing a repository-specific excerpt will ultimately involve some programmable logic specified in the repository, so I might defer that part initially. And it might end up being something like: you specify a rhai script that we'll run in a safe context, and for a limited duration, and give you access to data from the job so you can produce your own summary.
It would be handy if the GitHub summary for a failed run provided a button to automatically file an issue on the repo. I hope this would make it much easier for people to file issues for flaky tests in CI.
More concretely, on this screen: https://github.com/oxidecomputer/omicron/runs/20443797017
It'd be neat to have a button that would file an issue for this failure. Here's an example issue: https://github.com/oxidecomputer/omicron/issues/4590
The issue has:
cargo nextest
output? Or maybe there's already a parseable form of output?We might also consider a link to search for the same issue already having been filed.