Sudden extreme performance drop

PassiveLemon commented 1 day ago

I don't have exact times but yesterday I noticed that building onnxruntime was taking an abysmally long time. It didn't even finish in 8 hours when it would normally take around an hour and a half.

Today I noticed that this performance drop elsewhere (building wlx-overlay-s and some system packages).

I ran a little test, and though it's not very scientific, the numbers are significant: I started nh (nh os switch ~/Documents/GitHub/lemonix) and let it run until onnxruntime got to 10%. It took around a minute and 40 seconds.

Next I started a nixos-rebuild (sudo nixos-rebuild switch) and did the same. Onnxruntime got to 10% in around 25 seconds. My flake is symlinked to /etc/nixos/ so I don't need to supply the flake to this command.

I repeated this a few times and the average times are what I gave above.

My CPU has 12 threads. I set my nix settings to 11 cores and 1 max-job.

I do have some resource settings on the nix-daemon to reduce hanging when building but they were never an issue until yesterday.

systemd = {
  services = {
    "nix-daemon".serviceConfig = {
      Slice = "nix-daemon.slice";
      OOMScoreAdjust = 1000;
    };
  };
  slices."nix-daemon".sliceConfig = {
    CPUQuota = "80%";
    ManagedOOMMemoryPressure = "kill";
    ManagedOOMMemoryPressureLimit = "80%";
  };
};

viperML commented 1 day ago

Nh just calls nix build. You may try --no-nom but I don't see how it would influence it

PassiveLemon commented 1 day ago

--no-nom doesn't appear to change anything other than unprettying the output. I'm not sure how this could even be caused. Definitely a weird issue.

I ran another test for much longer (to reach 122% on a different program) nh: 4.5 minutes nh --no-nom: 4.5 minutes nixos-rebuild: 1 minute

viperML / nh

Sudden extreme performance drop #159