vcunat
commented
1 year ago For a stop-gap workaround I'd probably mark these two steps with big-parallel tag, which should bring back reliability at the expense of some idle CPUs.
These machines run 2 jobs only and have 64 GiB of physical RAM. (I hope we always get the same amount of RAM or larger.)
Background
The
tarballjob (e.g. in this jobset; code) does various evaluation checks on nixpkgs by calling nix.It's two derivations (linearized) using just single thread (maybe except nix's GC). However, some steps require huge amount of RAM, which causes practical issues, also frequently on current hydra.nixos.org.
RAM consumed now
🥇
checking eval-release.nixstep is far the worst, max. around 36 GiB, as measured bytimepackage+command.🥈
generating packages.jsonstep gets around 18 GiB.Others use around 10 GiB or less, which seems... bearable.
Running in practice
Given enough RAM, the build takes 2x 10-15 minutes (roughly). If not, result depends on your machine. The current builders on hydra.nixos.org usually manage to keep CPUs relatively busy with other work, but jobs that "touch lots of RAM or disk" can get slowed down extremely due to (un)swapping and time out – mainly the
tarballjobs themselves but not exclusively.I believe that strict evaluation checks are nice to have, even as blocker for channels, so I'd prefer to preserve at least similar level of strictness.