Build fails with 8GB space on /tmp - No space left on device

davidak commented 5 years ago

Issue description

I'm trying to rebuild with staging branch.

[root@imac:~]$  df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        802M     0  802M   0% /dev
tmpfs           7.9G  2.1M  7.9G   1% /dev/shm
tmpfs           4.0G  7.0M  4.0G   1% /run
tmpfs           7.9G  360K  7.9G   1% /run/wrappers
/dev/sda4       458G  134G  300G  31% /
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs           7.9G   17M  7.9G   1% /tmp
/dev/sda1       197M  174M   24M  89% /boot
tmpfs           1.6G   88K  1.6G   1% /run/user/1000
tmpfs           1.6G  4.0K  1.6G   1% /run/user/78

But the rebuild fails:

[ 48%] Building HexagonGenDAGISel.inc...
  CXX      menuattributeiter.lo
varobj.c:2539:1: fatal error: error writing to /build/ccE65Is0.s: No space left on device
 }
 ^
/build/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp:243:1: fatal error: error writing to /build/ccUCp0u7.s: No space left on device
 }
 ^
compilation terminated.
compilation terminated.
make[2]: *** [Makefile:1622: varobj.o] Error 1
make[2]: Leaving directory '/build/gdb-8.2.1/gdb'
make[1]: *** [Makefile:8825: all-gdb] Error 2
make[1]: Leaving directory '/build/gdb-8.2.1'
make: *** [Makefile:850: all] Error 2
make[2]: *** [lib/ExecutionEngine/CMakeFiles/LLVMExecutionEngine.dir/build.make:89: lib/ExecutionEngine/CMakeFiles/LLVMExecutionEngine.dir/GDBRegistrationListener.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:5574: lib/ExecutionEngine/CMakeFiles/LLVMExecutionEngine.dir/all] Error 2
make: *** [Makefile:152: all] Error 2
builder for '/nix/store/l6zb0p9x0qh61c2z01vjnpbhv0p0q93l-gdb-8.2.1.drv' failed with exit code 2
note: build failure may have been caused by lack of free disk space
building '/nix/store/kgfvxx6g7vhbihb9wpz9j2ha5zjlywh1-gstreamer-0.10.36.drv'...
cannot build derivation '/nix/store/5717l1jaw5lgavps9h2gd3g6ppbw631s-rustc-1.31.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9wd6hx622jlg109crglkwcq6dzn7psh4-librsvg-2.44.11.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/j8h056pxlalrgxyyd6b2pmd8l74wv46h-gst-plugins-bad-1.14.4.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ma311sr4vz5bb9kn7ljd49mfwqzhdsjg-wine-3.0.4.drv': 1 dependencies couldn't be built
building '/nix/store/zkpdcw0yyn94hd92qz222j052r45jpay-gts-0.7.6.drv'...
cannot build derivation '/nix/store/wcckb16wd1sc3i41d761fqwd8dsv3f1l-playonlinux-4.3.4.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vnkwq8rkgwqpzgkfsr5kxfndw0hmkys6-system-path.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/gqd0jr4ljpszh8c63wcbk6bvwqx0r302-nixos-system-imac-19.03.git.319ebef.drv': 1 dependencies couldn't be built
error: build of '/nix/store/gqd0jr4ljpszh8c63wcbk6bvwqx0r302-nixos-system-imac-19.03.git.319ebef.drv' failed

Steps to reproduce

boot.tmpOnTmpfs = true;

rebuild on staging, building all the things

Workaround

mount -o remount,size=15G /tmp

Technical details

system: "x86_64-linux"
host os: Linux 4.19.16, NixOS, 19.03.git.8291376 (Koi)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.2
channels(root): "nixos-18.09.1829.0396345b794, nixos-hardware, nixos-unstable-19.03pre164715.201d739b0ff"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

neilmayhew commented 2 years ago

I have this in configuration.nix:

{
  boot = {
    tmpOnTmpfs = true;
    tmpOnTmpfsSize = "10G";
  };
}

Then,

$ systemctl cat tmp.mount
# /etc/systemd/system/tmp.mount
[Unit]

[Mount]
Options=mode=1777,strictatime,rw,nosuid,nodev,size=10G
Type=tmpfs
What=tmpfs
Where=/tmp

$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            10G   36M   10G   1% /tmp

The value of size is an option that's interpreted by the tmpfs filesystem driver and documented in the tmpfs man page. Percentage values are relative to available RAM, so 50% allows /tmp to use up to 50% of your RAM.

MikiVanousek commented 2 years ago

@neilmayhew

I have this in configuration.nix:
{
  boot = {
    tmpOnTmpfs = true;
    tmpOnTmpfsSize = "10G";
  };
}
...

The value of size is an option that's interpreted by the tmpfs filesystem driver and documented in the tmpfs man page. Percentage values are relative to available RAM, so 50% allows /tmp to use up to 50% of your RAM.

This works for me indeed. Thanks :+1:. I was confused by the docs for the option tmpOnTmpfsSize. I think they are incomplete/incorrect. It is unclear what 100% means, and the option to specify size directly (as you did) is omitted. Do you agree? Should we submit a pull request for the change?

MikiVanousek commented 2 years ago

@MikiVanousek what's the content of /etc/fstab and the output of mount? Also boot.tmpOnTmpfs, boot.tmpOnTmpfsSize override this via a different mechanism and will probably conflict if both fileSystems."/tmp" and boot.tmpOnTmpfs is used. Output of ls /etc/systemd/system/*.mount would also be useful.

@zseri It indeed conflicted with tmpOnTmpfsSize. Thanks for your help :+1:!

neilmayhew commented 2 years ago

I was confused by the docs for the option tmpOnTmpfsSize. I think they are incomplete/incorrect. It is unclear what 100% means, and the option to specify size directly (as you did) is omitted. Do you agree? Should we submit a pull request for the change?

I don't agree, actually. The docs say,

Percentage is defined by systemd.

which is correct. The value is put straight into tmp.mount and not interpreted by NixOS at all, so it would be inappropriate to duplicate systemd's documentation here as it may not stay up to date.

Systemd's documentation for Options says,

Mount options to use when mounting

so these values aren't interpreted by systemd either. They're given to mount and finally interpreted by the tmpfs driver. The documentation for that says,

The tmpfs filesystem supports the following mount options:

size=bytes Specify an upper limit on the size of the filesystem. [etc.]

I don't think the trail to that is particularly hard to follow, but perhaps it would save some inconvenience if the documentation was changed to say:

Size of the tmpfs mounted on /tmp. Given to systemd.mount(5) as-is and ultimately passed to tmpfs(5).

edrex commented 1 year ago

An optional attribute buildSpaceRequired with an example value of "16G" (compatible with GNU dd units) for packages that require a lot of space to build would help save wasted builds in the future.

At least it would be great if we could establish some good upper bounds for memory required the few massive builds (chromium, firefox, what else?).

The default 50% on my 16GB laptop results in an 8GB /tmp, which isn't enough for chromiumDev. @cole-h said 16GB was sufficient. Trying with

> sudo mount -o remount,size=80% /tmp
> mount|grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=12888740k)

NixOS / nixpkgs