Deployment with nixops starts 32 build processes

[tom-kuca]

I'm trying to deploy nixos VPS from another VPS with nixops. I run multiple times into an OOM killer. The build tool spawns a few dozens of processes, be it rustc (ripgrep) or webpack (gitlab assets) just before it's shut down. I managed to compile ripgrep by assigning more resources (2 GB -> 4 GB), webpack sitll fails.

The command that fails:

/tmp/nix-build-gitlab-assets-13.0.9.drv-1/source/node_modules/.bin/webpack \
 --config /tmp/nix-build-gitlab-assets-13.0.9.drv-1/source/config/webpack.config.js --bail 

Deploy command:

nixops deploy -d simple --max-jobs 1 --cores 1

VPS #16200 has 4 GB, it runs on node15.prg. dmesg show there were 32 node processes when OOM killer killed webpack.

32 seems supsicious, I think the build tools get information that they should start 32 instances (maybe from host machine?). They do and OOM killer is spawned soon because there isn't enough memory on VM.

I was able to reproduce it multiple times, always with the same result.

Update 1: ~manual compilation passes:~ (it''s not full compilation, probably some env is missing)

cd /tmp/nix-build-gitlab-assets-13.0.9.drv-2/source
nix-shell /nix/store/rssbqrlxwp6ds15lnz6jww6xvr5vabp8-gitlab-assets-13.0.9.drv  
tmp/nix-build-gitlab-assets-13.0.9.drv-2/source/node_modules/.bin/webpack \
  --config /tmp/nix-build-gitlab-assets-13.0.9.drv-2/source/config/webpack.config.js --bail 

Update 2:

• On source VPS /proc/cpuinfo shows 32 cores on the VPS
• On target VPS, /proc/cpuinfo shows only 8 cores (as expected). The target VPS is also a nixos on node15.prg.
• On target VPS in container (created via nixops), /proc/cpuinfo shows 32 cores.

I don't know what's the difference between source VPS and target VPS, I would expect the same result on both.

[aither64]

Well, 32 cores on the sources VPS (called builder) was a bug in vpsAdmin which somehow let you disable CPU limit altogether... I've set it again, so you can give it another go...

But the issue is that nproc will report number of CPUs from the host and not the VPS, so wrong /proc/cpuinfo might not have caused this, it depends which way to check available CPUs the application uses.

Oh and when you're using nixops, the target VPS doesn't matter at all. Everything is built on the source and then only copied to the target, so you might ignore the target when you have problems with building.

[tom-kuca]

My account was allowed to bypass the limits for some unknown reason. It was resolved in ticket #34044.

I would expect I can overwrite nproc with --cores or NIX_BUILD_CORES, so it may not be a problem if that one is not correct. webpack shouldn't care, I will give it a try. For number of cpus webpack uses something like:

var _os = require("os");
console.log('os', _os)
console.log('cpus', _os.cpus());

strace shows that it reads /proc/cpuinfo.

[aither64]

It's not related to #34044, this was a new one https://github.com/vpsfreecz/vpsadmin/commit/53ad85a1de44fdbe6d5b26948535fcc0a4f9c357

Yep, it always depends on the build system if it uses autodetection or if it can be configured... on NixOS, you can set options nix.maxJobs and nix.buildCores on the source VPS to have it persistent.

[tom-kuca]

I confirm webpack now spawns only 4 nodes instead of 32. Closing. Thanks for the fix and hints.

The webpack compilation is still killed by OOM, but now it looks like a greedy webpack.