Open Mindavi opened 2 years ago
mweinelt
commented
2 years ago @mweinelt We need a timestamp when things are marked broken -> is a commit not good enough with git blame? It has a timestamp by itself. Anyway, should also be easy to add a date after the broken =
.
The idea here was to reuse the way we handle it for aliases. I'm open to other solutions of course.
Mindavi
commented
2 years ago I missed a thing here:
I think we could do the same as with aliases here, with a clear timestamp that's easily greppable it could make it easier to do cleanups, I guess.
kira-bruneau
commented
2 years ago Oooh, I love this idea :heart:! To avoid duplication, would it make sense to base nixpkgs-mark-broken off of the json output of hydra-check?
Also, what should we do in the rare cases where the package can build locally, but fails to build in hydra for whatever reason. For example, python3Packages.pygls builds fine locally on Darwin, but fails in hydra: https://hydra.nixos.org/build/186162604/nixlog/1. I'm not quite sure how to fix this :confused:.
What if a build is broken on the 4 platforms hydra typically builds (x86_64-{linux,darwin}, aarch64-{linux,darwin}? Should it mark meta.broken = true; to mark broken for all platforms? I think it makes sense since we otherwise cannot track it it's healthy anyway.
I think this is reasonable, this is better than the situation where you manually mark it as broken = true when you only tested one platform. If some obscure package only builds on an architecture we don't run hydra on, it should only list that platform as a supported platform.
What if someone is just fixing the build and then a mark-broken PR is merged? -> I'd say that that's an acceptable risk and that it's easy to unmark a package broken again, especially with a good commit message giving some background about when and how.
I think that should be ok, this could happen anyway with manually marking a package as broken, right? Maybe it could be useful to have a bot that checks when a PR touches a package that has been marked as broken after the merge base?
cdepillabout
commented
2 years ago Not that I think it will be that helpful for the general case, but we have an elaborate system to automatically mark Haskell packages as broken based on the build results from Hydra.
We have a document describing how we maintain the Haskell stuff in Nixpkgs (although only part of it is about marking things as broken): https://github.com/NixOS/nixpkgs/blob/35e26e2fea230ec7325e1811faa86b6072133927/pkgs/development/haskell-modules/HACKING.md
Here's an example build status report where you can see some Haskell packages (that have maintainers) have failed to build, so they need to either be fixed or marked broken: https://github.com/NixOS/nixpkgs/pull/180512#issuecomment-1178522352
After using this system for about a year or so, the biggest pain point is that Hydra is quite flaky, especially for darwin and aarch64-linux builds. It is very common for us to see build failures on darwin and aarch64-linux that are fixed just by restarting the build on Hydra. It is very frustrating (especially as someone who doesn't use other of those systems).
Mindavi
commented
2 years ago Also, what should we do in the rare cases where the package can build locally, but fails to build in hydra for whatever reason. For example,
python3Packages.pyglsbuilds fine locally on Darwin, but fails in hydra: https://hydra.nixos.org/build/186162604/nixlog/1. I'm not quite sure how to fix this 😕.
Maybe it makes sense to set hydraPlatforms = [ ]; for those? Or at least disable hydra builds for the platforms it's broken for only on hydra? Of course it would be better to fix them but I guess sometimes that's not feasible.
kira-bruneau
commented
2 years ago Oh yeah that could work! It might be a bit awkward for pygls though because it builds on Linux, but I guess we could just use something like hydraPlatforms = lib.subtractLists lib.platforms.darwin platforms;, or maybe introduce something like hydraBroken = stdenv.isDarwin.
Mindavi
commented
2 years ago After using this system for about a year or so, the biggest pain point is that Hydra is quite flaky, especially for darwin and aarch64-linux builds. It is very common for us to see build failures on darwin and aarch64-linux that are fixed just by restarting the build on Hydra. It is very frustrating (especially as someone who doesn't use other of those systems).
Thanks for mentioning your tools!
I think the way I'd like to address flakiness is to keep track of how long builds are broken and not marking one-offs as broken. If a build consistently fails for > x time, only then it should be marked broken, I think.
Mindavi
commented
1 year ago See https://github.com/NixOS/nixpkgs/pull/203997 for a first PR related to this. Tool is very much WIP still, but hopefully you find it a good start.
I also generated a list:
Intro
I'm considering building some kind of tool that can automatically mark a package broken if it fails to build.
I'd like some feedback as to how such a tool should work and what it should do exactly.
Requirements
meta.brokenfor the platforms it's supported formeta.brokenfor the platforms where the build fails, e.g. iflinuxis supported andaarch64-linuxbuilds butx86_64-linuxdoesn't, the result should bemeta.broken = (stdenv.isLinux && stdenv.isx86_64);(or something that has the same meaning)Open questions
meta.broken = true;to mark broken for all platforms? I think it makes sense since we otherwise cannot track it it's healthy anyway.Feedback (from matrix)
git blame? It has a timestamp by itself. Anyway, should also be easy to add a date after thebroken = <platform>.Initial draft
FYI, I already made an initial implementation that can make a list of all packages on trunk with their build status and can then filter the broken packages out of that list: https://github.com/Mindavi/nixpkgs-mark-broken. Doesn't have any documentation yet, still hacking on it. But I thought it might be useful to discuss with the community if they have some ideas/opinions before submitting any PRs with this.
Related issues