Mic92 / nixpkgs-review

Review pull-requests on https://github.com/NixOS/nixpkgs
MIT License
350 stars 59 forks source link

Report outputs that are larger than Hydra's size limit #398

Open mschwaig opened 2 months ago

mschwaig commented 2 months ago

Issue

Hydra has a size limit for outputs, which means that outputs larger than 3 GB are not cached and the Hydra job that builds them will fail.

Proposed Solution

I think nixpkgs-review should report packages that build successfully, but have such derivations in their runtime closure as failing.

Example Occurrence

As an example, this could have caught my change here https://github.com/NixOS/nixpkgs/pull/287846, which made a part of ROCm larger than Hydra's size limit.

Related issues

This issue is different from https://github.com/Mic92/nixpkgs-review/issues/36, because it is about the limits Hydra imposes on individual derivations, and not about keeping an eye on the overall size of the closure.

Current state /Alternatives

Right now a rough check of this limit can be done manually on the results of nixpkgs-review like this:

mschwaig@hydralisk ~/.c/n/pr-298388-9> cd results/
mschwaig@hydralisk ~/.c/n/p/results> du -h --apparent-size --max-depth 0 */ | grep G
3.7G  rocmPackages.composable_kernel/
1.6G  rocmPackages.miopen/
3.1G  rocmPackages.rocblas/
2.1G  rocmPackages.rocfft/
1.5G  rocmPackages.rocsolver/
1.2G  rocmPackages.rocsparse/

original comment

It's a bit tricky because the --apparent-size flag is really important on file systems with compression like ZFS, and it takes some effort to find out how Hydra actually measures size and what the configured limit is.

Mic92 commented 1 month ago

Sounds good. We can skip those warnings however for unfree software.