OOM during flake check on nix >= 2.10

[kjeremy]

Describe the bug

I have a flake (unfortunately proprietary) that has a lot of modules. Unfortunately nix flake check OOMs on nix >= 2.10 by wanting over 16GB of memory.

nix flake check -vvv ends with the following before dying:

instantiated 'python-namespaces-hook.sh' -> '/nix/store/7gxgqb5qgspcir93z18456911sa9yvxb-python-namespaces-hook.sh.drv' instantiated 'python-setup-hook.sh' -> '/nix/store/1mhy0bg06r9abj0rdj333dw0m4hv6rdg-python-setup-hook.sh.drv' instantiated 'Python-3.10.9.tar.xz' -> '/nix/store/zfj2940j78xn8470wzg1ayrvm29lpb0d-Python-3.10.9.tar.xz.drv' instantiated 'python3-3.10.9' -> '/nix/store/fgan9skx3j0m11ay92czb4qffnfjylvv-python3-3.10.9.drv' instantiated 'python-setup-hook.sh' -> '/nix/store/1mhy0bg06r9abj0rdj333dw0m4hv6rdg-python-setup-hook.sh.drv' instantiated 'Python-3.10.9.tar.xz' -> '/nix/store/zfj2940j78xn8470wzg1ayrvm29lpb0d-Python-3.10.9.tar.xz.drv' instantiated 'python3-3.10.9' -> '/nix/store/fgan9skx3j0m11ay92czb4qffnfjylvv-python3-3.10.9.drv' instantiated 'python-setup-hook.sh' -> '/nix/store/1mhy0bg06r9abj0rdj333dw0m4hv6rdg-python-setup-hook.sh.drv' instantiated 'Python-3.10.9.tar.xz' -> '/nix/store/zfj2940j78xn8470wzg1ayrvm29lpb0d-Python-3.10.9.tar.xz.drv' instantiated 'python3-3.10.9' -> '/nix/store/fgan9skx3j0m11ay92czb4qffnfjylvv-python3-3.10.9.drv' instantiated 'python-setup-hook.sh' -> '/nix/store/1mhy0bg06r9abj0rdj333dw0m4hv6rdg-python-setup-hook.sh.drv' instantiated 'Python-3.10.9.tar.xz' -> '/nix/store/zfj2940j78xn8470wzg1ayrvm29lpb0d-Python-3.10.9.tar.xz.drv' instantiated 'python3-armv7l-unknown-linux-gnueabihf-3.10.9' -> '/nix/store/gd77z5il9afm220g94qlzhl49apkjmr6-python3-armv7l-unknown-linux-gnueabihf-3.10.9.drv' instantiated 'python-setup-hook.sh' -> '/nix/store/1mhy0bg06r9abj0rdj333dw0m4hv6rdg-python-setup-hook.sh.drv' instantiated 'Python-3.10.9.tar.xz' -> '/nix/store/zfj2940j78xn8470wzg1ayrvm29lpb0d-Python-3.10.9.tar.xz.drv' instantiated 'python3-3.10.9' -> '/nix/store/fgan9skx3j0m11ay92czb4qffnfjylvv-python3-3.10.9.drv'

Is there any information I can get to help debug this?

Expected behavior

nix flake check succeeds.

nix-env (Nix) 2.13.1

Additional context

Add any other context about the problem here.

Priorities

Add :+1: to issues you find important.

[kjeremy]

Will recheck once #7811 is in

[Et7f3]

And now is it improved or I need to find more leaks ?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/unexpected-massive-memory-usage-when-evaluating-a-derivation-from-a-cross-compiling-nixpkgs/51039/2

[kjeremy]

I was just playing around with the example in that discourse post:

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs?rev=c374d94f1536013ca8e92341b540eba4c22f9c62";

  outputs = {
    self,
    nixpkgs,
  }: let
    # pkgsSet is written in a way that it also works as a non-flake
    # default.nix without modifications, issue persists in non flake context
    pkgsSet = {
      nixpkgs ? <nixpkgs>,
      localSystem ? builtins.currentSystem,
    }: let
      # ensure we're cross compiling
      crossSystem = {
          "x86_64-linux" = "aarch64-linux";
          "aarch64-linux" = "x86_64-linux";
        }.${localSystem};

      pkgs = import nixpkgs {
        system = localSystem;
      };

      pkgsCross = import nixpkgs {
        inherit crossSystem localSystem;
      };

      mkLargePython = pkgs:
        pkgs.python3.withPackages (ps:
          builtins.attrValues {
            inherit (ps) numpy matplotlib requests pandas;
          });
    in {
      python = mkLargePython pkgs; # native, no problem
      hello = pkgsCross.hello; # small cross, no problem
      pythonCross = mkLargePython pkgsCross; # big cross, ooms during evaluation
    };
  in {
    packages = nixpkgs.lib.genAttrs ["x86_64-linux" "aarch64-linux"] (system:
      pkgsSet {
        inherit nixpkgs;
        localSystem = system;
      });
  };
}

It doesn't OOM on my machine (32GB of RAM) but here are some results:

[jkolb@nixos:~/eval-bomb]$ command time -v nix eval --no-eval-cache .#python
warning: Git tree '/home/jkolb/eval-bomb' is dirty
«derivation /nix/store/ygx8zgvj0p7j4d64jpqi15sgq5lbxx9v-python3-3.12.4-env.drv»
    Command being timed: "nix eval --no-eval-cache .#python"
    User time (seconds): 0.95
    System time (seconds): 0.42
    Percent of CPU this job got: 74%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.84
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 238076
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 55519
    Voluntary context switches: 5633
    Involuntary context switches: 50
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

[jkolb@nixos:~/eval-bomb]$ command time -v nix eval --no-eval-cache .#pythonCross
warning: Git tree '/home/jkolb/eval-bomb' is dirty
«derivation /nix/store/2aklmw7wb84pjl4bg2llnnzn4hklyqll-python3-aarch64-unknown-linux-gnu-3.12.4-env.drv»
    Command being timed: "nix eval --no-eval-cache .#pythonCross"
    User time (seconds): 18.14
    System time (seconds): 3.76
    Percent of CPU this job got: 92%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:23.77
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 7624692
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 1902293
    Voluntary context switches: 19729
    Involuntary context switches: 778
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

The memory/time differences are pretty dramatic.

python	pythonCross
1.84 secs	23.77 secs
232 MB	7,445 MB

[Mic92]

If it's actually cross-compiling that causes your slow down, I would be surprised this actually significantly got slowed down in new nix versions. Do you have a before/after for that using different nix version? Especially memory usage. I think this is rather an architectural problem in nixpkgs than an issue nix can solve easily.

[kjeremy]

This is just eval. It didn't appear to be cross compiling anything but I can check on Monday.

[kjeremy]

@Mic92 there doesn't appear to be any cross-compilation going on according to htop. If I nix eval --no-eval-cache -vv .#pythonCross it is just generally slower processing files and hangs for many seconds on a number of them.

[Mic92]

I mean evaluation for cross compiled packages is just slow because of architectural issues in nixpkgs because it needs to evaluate derivations for different targets.

[kjeremy]

That makes some sense to me but the discrepancy is just massive. Should I file a bug with nixpkgs?

[kjeremy]

@Mic92 I filed the bug NixOS/nixpkgs#338231