Find ways to reduce RAM usage of nix-index

[suhr]

summary

To compute the mapping from the nixpkgs attribute name (ex: nixpkgs.hello) to the store path (ex: /nix/store/13208adkjlqwe...-hello), nix-index needs to evaluate nixpkgs. The current implementation uses nix-env -qaP --out-path --xml, which has high RAM usage since it needs to evaluate all derivations in nixpkgs.

Some possible ways to improve this within nix-index itself:

• offer an option to turn off the store path -> attribute mapping (resulting in a less helpful index, but one that can be built more easily)
• try to gather the mapping from the hydra job names (need to find a way to do this without causing too much load on the Hydra API)

---

original issue

It runs an instance of nix-env which eats all the RAM available in my system.

[bennofs]

This issue cannot easily be fixed in nix-index itself. The evaluation of the Hydra jobset is known to consume a lot of RAM.

[anka-213]

How much RAM does it use currently?

Also, how much time does it usually take and what size can I expect the final index to have? It might be nice to add some estimates of this to the readme.

[tilpner]

@anka-213

00:00:01 + querying available packages
...
00:21:40 + wrote index of 28,717,754 bytes
nixpkgs-index.service: Succeeded.
nixpkgs-index.service: Consumed 3min 43.981s CPU time, received 212.2M IP traffic, sent 32.3M IP traffic.

[milahu]

How much RAM does it use currently?

at least 1.3 GB

in ksysguard it looked more like 3 GB with 8 GB of RAM and no swap, i ran into out-of-memory (other apps took around 3 GB)

mprof (simple)

``` nix-shell -p python38Packages.memory_profiler nix-index \ --run 'time mprof run nix-index' nix-shell -p gnuplot --run gnuplot <<< ' set title "mprof nix-index"; set xlabel "Minutes"; set ylabel "MB RAM"; set term png; set output "mprof.png"; set key autotitle columnhead; unset key; # ignore first line first=0; offset=0; shift(x)=(offset=(first==0)?x:offset,first=1,x-offset); # relative time plot "mprofile_20210420003255.dat" using (shift($3/60)):2 w lp;' ``` 

valgrind massif (fancy)

recompile `nix-index/default.nix` with ``` buildType = "debug"; ``` and run ``` valgrind --tool=massif nix-index ls massif.out.* massif-visualizer massif.out.* ``` that 800 MB baseline looks wasteful

tempfiles

on the first run, nix-index stores 800 MB of tempfiles in /run/user/1000/file_listing.json* until /run/user/1000 is 100% full nix-index runs for 20 minutes, with lots of

Error: fetching the file listing for store path '/nix/store/w2qimbfmzl4p9xw3ahx7vqm5kz8imca4-flat-remix-icon-theme-20200116' failed
Caused by: response to GET 'http://cache.nixos.org/w2qimbfmzl4p9xw3ahx7vqm5kz8imca4.ls' failed to parse
Caused by: trailing characters at line 1 column 10146950

probably caused by the no space left on tmpfs

on the second run, nix-index stores 3.5 GB of tempfiles in /tmp/file_listing.json*

all these tempfiles are not deleted on finish

[turion]

Any ideas to improve this?

[bennofs]

The tempfiles are definitely a bug. They should only be kept if there is an error (to aid debugging), but perhaps there should be a limit on the number of errors to store (maybe even as a flag, which could default to 0)

[tobiasBora]

I also hit this issue on raspberry pi: I can't even compile the system (quite basic, just xfce and a few basic packages) as it runs out of memory. I use flake via nixos-rebuild switch --flake . to install the system, any idea if a fix for this issue would also help flake? Or should I open another issue?

[milahu]

on raspberry pi: I can't even compile the system ... as it runs out of memory

cross compile

[tobiasBora]

Yeah, well for now I can create a 2G swapfile and it seems to be good enough… (1G works sometimes… but not always.) The problem with cross-compilation is that it won't be cached. Also, people also recommended setting the raspberry pi as a remote builder (compilation does not take RAM apparently), or using binfmt to fake an Aarch64 system on a x86_64 system… but I don't really like this idea as it means that I need another computer, that it's harder to work in team (binfmt must be enabled on all systems… and the raspberry pi may not be accessible because of firewalls).

In any case I guess it's cool to solve this problem, burning GB of RAM just to evaluate nixpkgs is not great, and limits its applications on low-end devices/old devices (yeah I have some old computers at home that have very little RAM but that work great for simple servers). Also, I remember having issues with nix being really slow on another computer (like maybe one minute just to do a no-op nixos-rebuild switch) without any SSD: I think that the reason for that is that it tries to evaluate the whole nixpkgs repository and therefore needs to access the disk a lot… It would be much better to only evaluate what's needed.

[milahu]

were getting offtopic, this is nix-index, not nix

burning GB of RAM just to evaluate nixpkgs

please verify. i guess that building needs more ram

nix-index needs much ram, because it fetches a million files from cache.nixos.org

some more ideas: disable parallel building disable parallel fetching? log memory usage of all processes during nix-build

one problem for low-memory devices is that xz -d needs about 5x more memory than gzip -d and cache.nixos.org does not serve nar.gz files

``` curl https://cache.nixos.org/$(readlink -f $(which gimp) | cut -c12-43).narinfo StorePath: /nix/store/sqvlmp2dkrvyisi11dq6w6wdlbhyikfj-gimp-2.10.32 URL: nar/1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.xz Compression: xz FileHash: sha256:1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j FileSize: 19313388 NarHash: sha256:14gpflz0d8ng4zpyx9sm3g4z2p7a4zscdmrp3dph5l6ijkdyvmiv NarSize: 114638192 curl -I https://cache.nixos.org/nar/1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.xz HTTP/2 200 curl -I https://cache.nixos.org/nar/1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.gz HTTP/2 404 cd $(mktemp -d) nix-shell -p time wget https://cache.nixos.org/nar/1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.xz $(which time) -v xz -d -k *.xz 2>&1 | grep "Maximum resident set size" Maximum resident set size (kbytes): 10592 gzip -k *.nar rm *.nar $(which time) -v gzip -d -k *.gz 2>&1 | grep "Maximum resident set size" Maximum resident set size (kbytes): 1836 du -sh * 110M 1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar 32M 1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.gz 19M 1848svxsb5yy8dnnw46rv6p4j01bzs11073w6r79wm37iqjiba2j.nar.xz ```

[socherbyc]

Just as a note: I have 16GB RAM and I'm running out of memory too :/

[Krutonium]

16GB + 4GB zswap + 8GB of swap and I ran out of memory.

[skull-squadron]

RES 15.1 of 24 GiB and nix-index just sits at + querying available packages for 10 minutes before it spews a bunch of shit to the console. + generating index: 10759 paths found :: 24841 paths and then it just sits around again without status. So poor resource usage and not good UX.

[Frontear]

Bumping this. I've been able to make a cache before, but suddenly having 12GB RAM + 18GB ZRAM apparently is insufficient and oom-killer fires even before it can finish querying all packages. This is with absolutely NO other processes (besides system daemons) being active and on a very minimal sway environment.

I'm suddenly wondering how I ever managed to run it because now it absolutely refuses to run, no matter how many times I run it. I'd probably have to change the OOM Score for it to work, but I imagine it'll hard freeze my system.

Output of free -m before nix-index:

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           11705         438       11084          67         182       10724
Swap:          17557         179       17377
[frontear@LAPTOP-3DT4F02 dotfiles]$ 

Entries from dmesg:

[ 1234.253242] nix-env invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[ 1234.253246] CPU: 0 UID: 1000 PID: 2742 Comm: nix-env Not tainted 6.11.0 #1-NixOS
[ 1234.253248] Hardware name: Dell Inc. Inspiron 14 5410 2-in-1/0TPG56, BIOS 2.23.0 11/13/2023
[ 1234.253249] Call Trace:
[ 1234.253251]  <TASK>
[ 1234.253254]  dump_stack_lvl+0x64/0x90
[ 1234.253258]  dump_header+0x44/0x1c0
[ 1234.253262]  oom_kill_process+0x104/0x210
[ 1234.253265]  out_of_memory+0x256/0x590
[ 1234.253266]  __alloc_pages_noprof+0xe27/0x1130
[ 1234.253270]  alloc_pages_mpol_noprof+0xd9/0x1f0
[ 1234.253273]  folio_alloc_mpol_noprof+0x14/0x40
[ 1234.253274]  vma_alloc_folio_noprof+0x6b/0xd0
[ 1234.253275]  do_swap_page+0xb78/0x10d0
[ 1234.253279]  ? update_load_avg+0x7e/0x7f0
[ 1234.253282]  ? __pte_offset_map+0x1b/0x190
[ 1234.253284]  __handle_mm_fault+0x86e/0x1060
[ 1234.253287]  handle_mm_fault+0x17f/0x2e0
[ 1234.253289]  do_user_addr_fault+0x177/0x6b0
[ 1234.253293]  exc_page_fault+0x71/0x160
[ 1234.253296]  asm_exc_page_fault+0x26/0x30
[ 1234.253298] RIP: 0033:0x7f953ed2876b
[ 1234.253317] Code: 39 e8 0f 82 51 fe ff ff 48 8b 38 48 83 ea 10 49 39 fe 77 09 49 39 ff 0f 87 f7 00 00 00 48 83 e8 10 48 39 ea 0f 82 2f fe ff ff <48> 8b 3a 49 89 c5 0f 18 82 00 ff ff ff 49 39 fe 77 c3 49 39 ff 76
[ 1234.253319] RSP: 002b:00007ffd14539240 EFLAGS: 00010202
[ 1234.253321] RAX: 00007f93de688e68 RBX: 00007ffd14539270 RCX: 0000000000000000
[ 1234.253322] RDX: 00007f93de688e70 RSI: 0000000000000000 RDI: 0000000000000000
[ 1234.253323] RBP: 00007f93de688a80 R08: 0000000000000000 R09: 00007f930e2d4000
[ 1234.253323] R10: 00000000000003f8 R11: 0000000000000000 R12: ffffffffffffff38
[ 1234.253324] R13: 0000000000000330 R14: 00007f91232d4820 R15: 00007f95408b904a
[ 1234.253327]  </TASK>
[ 1234.253328] Mem-Info:
[ 1234.253329] active_anon:2146345 inactive_anon:536190 isolated_anon:0
                active_file:2923 inactive_file:3403 isolated_file:0
                unevictable:8132 dirty:0 writeback:1
                slab_reclaimable:12959 slab_unreclaimable:23978
                mapped:6595 shmem:12061 pagetables:8050
                sec_pagetables:771 bounce:0
                kernel_misc_reclaimable:0
                free:16729 free_pcp:1230 free_cma:0
[ 1234.253332] Node 0 active_anon:8585380kB inactive_anon:2144760kB active_file:11692kB inactive_file:13612kB unevictable:32528kB isolated(anon):0kB isolated(file):0kB mapped:26380kB dirty:0kB writeback:4kB shmem:48244kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:4704kB pagetables:32200kB sec_pagetables:3084kB all_unreclaimable? no
[ 1234.253335] Node 0 DMA free:84kB boost:0kB min:84kB low:276kB high:468kB reserved_highatomic:0KB active_anon:8040kB inactive_anon:5144kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1234.253338] lowmem_reserve[]: 0 1426 11652 0 0
[ 1234.253340] Node 0 DMA32 free:8016kB boost:0kB min:8420kB low:27088kB high:45756kB reserved_highatomic:0KB active_anon:1275268kB inactive_anon:170232kB active_file:128kB inactive_file:732kB unevictable:24kB writepending:4kB present:1559724kB managed:1493600kB mlocked:0kB bounce:0kB free_pcp:3684kB local_pcp:3128kB free_cma:0kB
[ 1234.253343] lowmem_reserve[]: 0 0 10226 0 0
[ 1234.253345] Node 0 Normal free:58816kB boost:0kB min:59076kB low:190044kB high:321012kB reserved_highatomic:0KB active_anon:7301260kB inactive_anon:1969504kB active_file:11344kB inactive_file:13204kB unevictable:32648kB writepending:0kB present:10739712kB managed:10477716kB mlocked:11540kB bounce:0kB free_pcp:1236kB local_pcp:472kB free_cma:0kB
[ 1234.253348] lowmem_reserve[]: 0 0 0 0 0
[ 1234.253350] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 84kB
[ 1234.253356] Node 0 DMA32: 48*4kB (UE) 15*8kB (UME) 12*16kB (UE) 10*32kB (UME) 7*64kB (UE) 8*128kB (UE) 7*256kB (UME) 3*512kB (UME) 2*1024kB (M) 0*2048kB 0*4096kB = 7672kB
[ 1234.253364] Node 0 Normal: 494*4kB (UM) 484*8kB (U) 191*16kB (UM) 420*32kB (UME) 220*64kB (UME) 79*128kB (UME) 20*256kB (UME) 13*512kB (ME) 0*1024kB 0*2048kB 0*4096kB = 58312kB
[ 1234.253372] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1234.253373] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1234.253374] 21010 total pagecache pages
[ 1234.253375] 72 pages in swap cache
[ 1234.253375] Free swap  = 15157244kB
[ 1234.253376] Total swap = 17978876kB
[ 1234.253377] 3078857 pages RAM
[ 1234.253377] 0 pages HighMem/MovableOnly
[ 1234.253377] 82188 pages reserved
[ 1234.253378] 0 pages cma reserved
[ 1234.253378] 0 pages hwpoisoned
[ 1234.253379] Tasks state (memory values in pages):
[ 1234.253379] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[ 1234.253385] [    531]     0   531    11851     1014       32      982         0    98304      256          -250 systemd-journal
[ 1234.253388] [    565]     0   565     8472     1095       64     1031         0    86016      704         -1000 systemd-udevd
[ 1234.253390] [   1049]   994  1049     3680     1063        0     1063         0    73728      160          -900 systemd-oomd
[ 1234.253392] [   1104]   154  1104    77643     1035        0     1035         0   118784      288             0 systemd-timesyn
[ 1234.253394] [   1119]     4  1119     3046     1131      160      971         0    61440      256          -900 dbus-daemon
[ 1234.253396] [   1130]     0  1130    97184      849        0      849         0   143360      224             0 thermald
[ 1234.253397] [   1158]     0  1158     3857     1027        0     1027         0    73728      256             0 systemd-logind
[ 1234.253399] [   1244]     0  1244    83851     2176      256     1920         0   159744      608             0 NetworkManager
[ 1234.253400] [   1328]   996  1328   171668      961       32      929         0   184320      352             0 nsncd
[ 1234.253402] [   1338]     0  1338   112047     1046        0     1046         0   135168      896             0 nix-daemon
[ 1234.253403] [   1514]     0  1514     3679     1357       32     1325         0    65536      384             0 wpa_supplicant
[ 1234.253405] [   1529]     0  1529      917      885      128      757         0    53248        0             0 greetd
[ 1234.253406] [   1675]     0  1675     2754     2738      299     2439         0    65536        0             0 greetd
[ 1234.253408] [   1938]  1000  1938     4973     1070       32     1038         0    81920      544           100 systemd
[ 1234.253409] [   1941]  1000  1941     4562      390       37      353         0    73728      320           100 (sd-pam)
[ 1234.253411] [   1958]  1000  1958   378203     1525      354     1140        31   622592     7360             0 sway
[ 1234.253412] [   1964]    28  1964    95449      787        0      787         0   122880      640             0 polkitd
[ 1234.253414] [   2021]  1000  2021     4095      599        0      599         0    69632      224             0 swaybg
[ 1234.253415] [   2030]  1000  2030   219419     2212      608     1572        32   393216     2080             0 .waybar-wrapped
[ 1234.253417] [   2032]  1000  2032    18538      486        0      486         0    69632       64             0 wl-clip-persist
[ 1234.253418] [   2034]  1000  2034     5724      602        0      602         0    77824     2400             0 papersway
[ 1234.253420] [   2036]  1000  2036     1070      464        0      464         0    45056       32             0 swayidle
[ 1234.253421] [   2043]  1000  2043     2529      485        0      485         0    57344       96             0 sh
[ 1234.253422] [   2045]  1000  2045     1415      610        0      610         0    49152       96           200 dbus-daemon
[ 1234.253424] [   2046]  1000  2046      895      447        0      447         0    45056       32             0 swaymsg
[ 1234.253425] [   2047]  1000  2047     5724      159        0      159         0    65536     2400             0 papersway
[ 1234.253426] [   2054]  1000  2054   119001      933        0      933         0   163840      704           200 .xdg-desktop-po
[ 1234.253427] [   2059]  1000  2059   133515      535        0      535         0   159744      160           200 .xdg-document-p
[ 1234.253429] [   2063]  1000  2063    77554      553        0      553         0   131072      128           200 .xdg-permission
[ 1234.253430] [   2070]  1000  2070      647      417        0      417         0    40960        0           200 fusermount3
[ 1234.253431] [   2073]  1000  2073   103796      725       32      688         5   196608     1152           200 .xdg-desktop-po
[ 1234.253433] [   2081]   995  2081     5139      522        0      522         0    57344       32             0 rtkit-daemon
[ 1234.253434] [   2084]  1000  2084    28987      724        0      724         0   114688     1120           200 pipewire
[ 1234.253435] [   2085]  1000  2085   120464      760        0      760         0   172032     1312           200 wireplumber
[ 1234.253437] [   2095]  1000  2095   132707     1070        0     1070         0   462848     6688             0 Xwayland
[ 1234.253438] [   2103]  1000  2103    87774      789        0      789         0   360448     2176           200 .xdg-desktop-po
[ 1234.253439] [   2148]  1000  2148   358436     1127      160      967         0   307200     1504             0 foot
[ 1234.253441] [   2149]  1000  2149     3838      932      295      637         0    73728      736             0 zsh
[ 1234.253442] [   2173]  1000  2173    39406      702       64      638         0    86016       32           200 gpg-agent
[ 1234.253443] [   2723]  1000  2723   278822     3357     1792     1565         0   335872     6976             0 .nix-index-wrap
[ 1234.253445] [   2742]  1000  2742  3335456  2673979  2671659     2320         0 26730496   646752             0 nix-env
[ 1234.253446] [   2762]     0  2762   112092      619        0      619         0   118784      938             0 nix-daemon
[ 1234.253447] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=nix-env,pid=2742,uid=1000
[ 1234.253458] Out of memory: Killed process 2742 (nix-env) total-vm:13341824kB, anon-rss:10686636kB, file-rss:9280kB, shmem-rss:0kB, UID:1000 pgtables:26104kB oom_score_adj:0