Open uri-canva opened 6 years ago
TL;DR: I think we should consider extending the set.
Current evaluation times: trunk
jobset is fairly fast now (100–250 s typically), trunk-combined
is a bit worse but still bearable (1000–2000), /cc @edolstra but it seems still OK to me. The OS/test part seem worst, so the impact probably won't be large. The Borg bot might actually feel this the most on CPU and RAM, /cc @grahamc, but we would be getting better coverage of its checks in exchange. nix-env -qa
also might be affected, but that doesn't seem an important use case, and we can differentiate anyway if need be (.recurseForDerivations
vs. .recurseForRelease
).
For builders themselves, x86_64-linux
certainly can sustain significantly more builds, but on the whole I'd expect building those sets to be cheap.
Some of the sets have returned in the meantime, e.g. python27Packages
, python36Packages
, ocamlPackages
, four variants of perl5??Packages
, so I guess it will be about considering particular cases.
BTW, evaluation was the problem according to the commit message, and that's unrelated to the epyc machine.
Ah I see, thanks for the clarification, I had misunderstood.
In my case it's specifically the nodePackages
recursion I'm interested in.
I'd be happy to put together a subset of R packages that are sufficiently popular to warrant serving binaries for them if it would be helpful.
Here's a list of the top 100 downloaded R packages from Rstudio's CRAN mirror (likely the best we can do to assess popularity). It would be really great if we had a binary cache for some/all of these as some take enough time to build locally to be annoying, especially on lower end machines (looking at you, Rcpp
). The numbers are the number of downloads during the last month off that mirror.
1 rlang 964812
2 Rcpp 890409
3 ggplot2 777626
4 dplyr 758201
5 data.table 738833
6 yaml 712247
7 tibble 702941
8 stringr 647555
9 pillar 644288
10 R6 627528
11 stringi 605047
12 curl 592074
13 cli 591365
14 digest 590345
15 crayon 590278
16 fansi 588222
17 htmltools 585499
18 glue 575145
19 purrr 569593
20 pkgconfig 568731
21 BH 549192
22 magrittr 544492
23 httr 544088
24 jsonlite 523928
25 utf8 519052
26 assertthat 510771
27 openssl 496001
28 knitr 489695
29 clipr 473974
30 colorspace 471929
31 shiny 460882
32 scales 459468
33 later 456398
34 mime 455129
35 evaluate 453942
36 httpuv 449545
37 htmlwidgets 448720
38 markdown 447110
39 plyr 442415
40 withr 440584
41 base64enc 438826
42 tidyselect 437980
43 reshape2 437623
44 xfun 433235
45 xtable 417969
46 readr 406843
47 prettyunits 405066
48 readxl 404383
49 munsell 400205
50 highr 398729
51 lazyeval 398257
52 RColorBrewer 397893
53 backports 394173
54 sys 393033
55 hms 387217
56 viridisLite 386345
57 promises 384669
58 zoo 384636
59 sourcetools 382674
60 devtools 382353
61 xml2 381892
62 rmarkdown 380086
63 tidyr 378568
64 askpass 378002
65 plogr 372225
66 gtable 363425
67 tinytex 361202
68 lubridate 360588
69 rstudioapi 357520
70 labeling 355730
71 callr 352961
72 progress 346663
73 processx 345723
74 ps 335529
75 dbplyr 332283
76 gridExtra 305707
77 broom 299361
78 haven 298729
79 WGCNA 298377
80 bindrcpp 293190
81 fs 291291
82 DBI 286358
83 tidyverse 279764
84 bindr 273468
85 cellranger 273152
86 rematch 269185
87 forcats 258780
88 generics 256182
89 rprojroot 251119
90 lme4 249779
91 git2r 248871
92 igraph 244045
93 whisker 237258
94 bitops 236360
95 xts 224842
96 usethis 221697
97 caTools 221628
98 modelr 215098
99 car 212805
100 rvest 212410
@NixOS/rfc-steering-committee (I don't know whom to ping about this): is it possible to get some kind of guidance about adding similar larger sets that may be a bit more expensive to some of the resources?
I suspect that the size of outputs will be the most expensive part to these. So, the approach would be to estimate the size (per one rebuild) and then decide? Possibly some whole sets might be cheap enough, e.g. if we restrict them via hydraPlatforms = [ "x86_64-linux" ];
Well, maybe the size of the JSON for https://nixos.org/nixos/packages.html might be also limiting, but that seems a secondary issue and we can control that set separately if need be.
Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:
Issue description
In d008513af25f264accdbd4f496bfe13803e28190 building of language specific packages was disabled on hydra. Now that faster hardware is available (https://grahamc.com/blog/an-epyc-nixos-build-farm) can they be re-enabled, or is it not worth it because they don't take very long to build locally? Even so, wouldn't that point be moot? If they don't take a long time to build locally they shouldn't take a long time to build on Hydra either.
Steps to reproduce
Technical details
Please run
nix-shell -p nix-info --run "nix-info -m"
and paste the results.