Running Nix as root ignores nix-damon's TMPDIR

Atemu commented 2 years ago

Describe the bug

A clear and concise description of what the bug is.

Nix builds ignore the nix-daemon's TMPDIR environment variable when run as root.

Steps To Reproduce

TMPDIR=/var/tmp nix-daemon
nix-build '<nixpkgs>' -A hello --check
sudo nix-build '<nixpkgs>' -A hello --check

Expected behavior

A clear and concise description of what you expected to happen.

Both builds should run in /var/tmp.

nix-env --version output

nix-env (Nix) 2.11.0

Additional context

Add any other context about the problem here.

This happens all the time when you try to run nixos-rebuild because that command obviously needs to run as root.
The kernel build notably uses more space than most people have space for in /tmp thanks to BTF. sudo nixos-rebuild switch with a custom kernel build is no bueno by default for the vast majority of systems. That's not a good UX.

Honestly, I don't see why /tmp should be the default. Sure, it can be a useful optimisation if you know your builds are all tiny but that's an assumption you cannot make for all builds in general.
/var/tmp is on a physical disk by default and therefore makes a much better default TMPDIR.

benaryorg commented 1 year ago

Nix builds ignore the nix-daemon's TMPDIR environment variable when run as root.

To add a specific use-case to this; this applies to deployments via colmena as it uses nix-store --realise as root for this.

Honestly, I don't see why /tmp should be the default. Sure, it can be a useful optimisation if you know your builds are all tiny but that's an assumption you cannot make for all builds in general. /var/tmp is on a physical disk by default and therefore makes a much better default TMPDIR.

To expand on this with a very clear rationale; if nix uses a separate directory I, as a sysadmin, can decide to mount either a tmpfs there, a different disk, a bindmount, whatever. If nix uses a directory that is shared with other data I do not have the ability to influence this.

Especially considering that setting boot.crashDump.enable will set custom config options and thus compile a fresh Linux kernel, which usually tends to exceed 8GiB in object files (especially during compression phase) this makes it unfeasible to run this in tmpfs on any machine with 16GiB of RAM or less, which I'd estimate is a very good chunk of machines out there.

Atry commented 1 year ago

This issue can result in failures in pkgs.writeShellApplication when running nix build under a dev shell with root user. I created a repository to reproduce the error in https://github.com/Atry/nix-tmpdir-bug-report

Reproducting step

Clone https://github.com/Atry/nix-tmpdir-bug-report.git
Open the work tree in VS Code
Install the Dev Containers extension https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers
Reopen it in a Dev Containers
Open a terminal
Run nix develop -c sh -c 'nix build'

Now you can see the following error:

error: builder for '/nix/store/29z9iy6zx9dvmh075dza7f7zhkmnvx2n-my-app.sh.drv' failed with exit code 1;
       last 2 log lines:
       > unpacking sources
       > variable $src or $srcs should point to the source
       For full logs, run 'nix log /nix/store/29z9iy6zx9dvmh075dza7f7zhkmnvx2n-my-app.sh.drv'.

Workarounds

Workaround 1: removing `"remoteUser": "root",` from the dev container configuration

https://github.com/Atry/nix-tmpdir-bug-report/blob/310e5f5e57b347d9e1ec0915aa1187bb063218c9/.devcontainer/devcontainer.json#L15

Workaround 2: unset TMPDIR

Run nix develop -c sh -c 'unset TMPDIR && nix build', then the build would succeed.

Workaround 3: Don't use `nix develop`

Directly run nix build, then the build would succeed.

doronbehar commented 1 year ago

This issue is very annoying - I ran into it now when performing a nixos-rebuild as root, and I have in my nixos config:

    boot.tmp.useTmpfs = true;
    # Use a tmpdir with unlimited space (as opposed to /tmp)
    systemd.services."nix-daemon".environment.TMPDIR = "/nix/tmp";
    systemd.tmpfiles.rules = [
      "d /nix/tmp 770 root nixbld"
    ];

obj-obj commented 1 year ago

if you run sudo env TMPDIR=/nix/tmp nixos-rebuild switch, it will use the correct tmpdir. Maybe just alias that for now?

Quant-ux commented 9 months ago

~~I primarily interact with the nix-daemon via nixos-rebuild and have used the alias workaround with success.~~

~~As of 23.11, the workaround no longer works because of https://github.com/NixOS/nixpkgs/pull/258571 unless NIXOS_SWITCH_USE_DIRTY_ENV is set; however, it is going to be removed in the future[1].~~

~~As it stands, to my knowledge, there is no future-proof, working workaround that applies to my use case.~~ ~~1. https://github.com/NixOS/nixpkgs/issues/258977~~

My issue was caused by a bad sudo configuration and after fixing it, the alias workaround works for me again.

thufschmitt commented 7 months ago

This is likely related to https://github.com/NixOS/nix/issues/10140: Running Nix as root will bypass the daemon altogether. Workaround: use NIX_REMOTE=daemon (or set store = daemon in your Nix config, and pass --store local to the daemon invocation in the systemd unit)

benaryorg commented 1 month ago

Note that Nix ≥2.22 (as well as Lix ≥2.91) have the new build-dir nix.conf setting.

NixOS / nix