search

NixOS / ofborg

@ofborg tooling automation https://monitoring.ofborg.org/dashboard/db/ofborg
https://ofborg.org
MIT License
232 stars 166 forks source link

report build failures as failures #675

Closed adamcstephens closed 4 months ago

adamcstephens commented 7 months ago

It is far too easy to miss clear failures when we report them as neutral. Not only does this not register them in GitHub visually as a failure, but once all checks have completed successfully GitHub collapses the checks UI and shows the PR as fully successfully checked.

We're already merging PRs that are failing checks, it's just not as visible. This change will at least provide visibility into those failing checks, to allow us to make better informed decisions when proceeding to merge anyway.

Note: I have little knowledge of what I'm doing, but this seems to be the correct place to change this.

alyssais commented 7 months ago

I don't think timed out should be a failure. That's something that happens extremely often (basically any PR targeting staging will time out on Darwin), and getting people used to merging despite a reported failure will cause alarm fatigue and make it more likely that real failures are ignored.

adamcstephens commented 7 months ago

Can we distinguish failures of the leaf OfBorg was trying to build from failures of dependencies? Another thing that comes up a lot especially on staging is failures of dependencies due to flaky tests or whatever, and those shouldn't block a PR being merged, so they shouldn't show up as failures.

I can't answer the first question as I don't know enough about the entire system. I don't think PRs are blocked regardless on failing ofborg checks.

alyssais commented 7 months ago

I don't think PRs are blocked regardless on failing ofborg checks.

Not on a technical level, but people get very nervous about ignoring a failure when merging, and in practice will very rarely do it without being very sure it's okay. That's a useful property that we should preserve, and if we lose it by accident, we won't get a second chance.

hraban commented 7 months ago

See related discussion here: https://discourse.nixos.org/t/can-we-make-ofborg-show-broken-tests-as-red/10717/2

adamcstephens commented 4 months ago

Happy to reopen at any time.

hraban commented 4 months ago

Any reason you closed it, @adamcstephens ? I thought this was a worthwile idea and I hope someone will pick it up someday!