Binary cache for interpreted languages packages

[uri-canva]

Issue description

In d008513af25f264accdbd4f496bfe13803e28190 building of language specific packages was disabled on hydra. Now that faster hardware is available (https://grahamc.com/blog/an-epyc-nixos-build-farm) can they be re-enabled, or is it not worth it because they don't take very long to build locally? Even so, wouldn't that point be moot? If they don't take a long time to build locally they shouldn't take a long time to build on Hydra either.

Steps to reproduce

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the results.

[vcunat]

TL;DR: I think we should consider extending the set.

Current evaluation times: trunk jobset is fairly fast now (100–250 s typically), trunk-combined is a bit worse but still bearable (1000–2000), /cc @edolstra but it seems still OK to me. The OS/test part seem worst, so the impact probably won't be large. The Borg bot might actually feel this the most on CPU and RAM, /cc @grahamc, but we would be getting better coverage of its checks in exchange. nix-env -qa also might be affected, but that doesn't seem an important use case, and we can differentiate anyway if need be (.recurseForDerivations vs. .recurseForRelease).

For builders themselves, x86_64-linux certainly can sustain significantly more builds, but on the whole I'd expect building those sets to be cheap.

Some of the sets have returned in the meantime, e.g. python27Packages, python36Packages, ocamlPackages, four variants of perl5??Packages, so I guess it will be about considering particular cases.

BTW, evaluation was the problem according to the commit message, and that's unrelated to the epyc machine.

[uri-canva]

Ah I see, thanks for the clarification, I had misunderstood.

In my case it's specifically the nodePackages recursion I'm interested in.

[jabranham]

I'd be happy to put together a subset of R packages that are sufficiently popular to warrant serving binaries for them if it would be helpful.

[jabranham]

Here's a list of the top 100 downloaded R packages from Rstudio's CRAN mirror (likely the best we can do to assess popularity). It would be really great if we had a binary cache for some/all of these as some take enough time to build locally to be annoying, especially on lower end machines (looking at you, Rcpp). The numbers are the number of downloads during the last month off that mirror.

---

1          rlang 964812
2           Rcpp 890409
3        ggplot2 777626
4          dplyr 758201
5     data.table 738833
6           yaml 712247
7         tibble 702941
8        stringr 647555
9         pillar 644288
10            R6 627528
11       stringi 605047
12          curl 592074
13           cli 591365
14        digest 590345
15        crayon 590278
16         fansi 588222
17     htmltools 585499
18          glue 575145
19         purrr 569593
20     pkgconfig 568731
21            BH 549192
22      magrittr 544492
23          httr 544088
24      jsonlite 523928
25          utf8 519052
26    assertthat 510771
27       openssl 496001
28         knitr 489695
29         clipr 473974
30    colorspace 471929
31         shiny 460882
32        scales 459468
33         later 456398
34          mime 455129
35      evaluate 453942
36        httpuv 449545
37   htmlwidgets 448720
38      markdown 447110
39          plyr 442415
40         withr 440584
41     base64enc 438826
42    tidyselect 437980
43      reshape2 437623
44          xfun 433235
45        xtable 417969
46         readr 406843
47   prettyunits 405066
48        readxl 404383
49       munsell 400205
50         highr 398729
51      lazyeval 398257
52  RColorBrewer 397893
53     backports 394173
54           sys 393033
55           hms 387217
56   viridisLite 386345
57      promises 384669
58           zoo 384636
59   sourcetools 382674
60      devtools 382353
61          xml2 381892
62     rmarkdown 380086
63         tidyr 378568
64       askpass 378002
65         plogr 372225
66        gtable 363425
67       tinytex 361202
68     lubridate 360588
69    rstudioapi 357520
70      labeling 355730
71         callr 352961
72      progress 346663
73      processx 345723
74            ps 335529
75        dbplyr 332283
76     gridExtra 305707
77         broom 299361
78         haven 298729
79         WGCNA 298377
80      bindrcpp 293190
81            fs 291291
82           DBI 286358
83     tidyverse 279764
84         bindr 273468
85    cellranger 273152
86       rematch 269185
87       forcats 258780
88      generics 256182
89     rprojroot 251119
90          lme4 249779
91         git2r 248871
92        igraph 244045
93       whisker 237258
94        bitops 236360
95           xts 224842
96       usethis 221697
97       caTools 221628
98        modelr 215098
99           car 212805
100        rvest 212410

[vcunat]

@NixOS/rfc-steering-committee (I don't know whom to ping about this): is it possible to get some kind of guidance about adding similar larger sets that may be a bit more expensive to some of the resources?

I suspect that the size of outputs will be the most expensive part to these. So, the approach would be to estimate the size (per one rebuild) and then decide? Possibly some whole sets might be cheap enough, e.g. if we restrict them via hydraPlatforms = [ "x86_64-linux" ];

Well, maybe the size of the JSON for https://nixos.org/nixos/packages.html might be also limiting, but that seems a secondary issue and we can control that set separately if need be.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.