Open davidak opened 5 years ago
Perhaps the issue is that /tmp
gets filled? I thing that is where /build
is mounted to.
/build/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp:243:1: fatal error: error writing to /build/ccUCp0u7.s: No space left on device
/build is a path in build sandbox, and it is located on /tmp, and 8GiB not being enough for an LLVM build is not completely unimaginable.
@jtojnar good idea!
After starting the build it is already 27% filled!
An optional attribute buildSpaceRequired
with an example value of "16G" (compatible with GNU dd
units) for packages that require a lot of space to build would help save wasted builds in the future.
Yes, it's actually /tmp
that is filled up. It goes up to 95% and then the build fails.
It would be helpful if it fails directly with a meaningful error message like @memberbetty suggested.
tar: firefox-60.4.0/layout/reftests/w3c-css/received/css-writing-modes/support/swatch-olive.png: Cannot write: No space left on device
tar: firefox-60.4.0/layout/reftests/w3c-css/received/css-writing-modes/support/mplus-1p-regular.woff: Cannot write: No space left on device
builder for '/nix/store/6qj2wsgnpp85ip00mnmcp68bihx7asrg-spidermonkey-38.8.0.drv' failed with exit code 2
note: build failure may have been caused by lack of free disk space
Meaningful... in the sense that nix should print it's about /tmp
?
@vcunat something like "PACKAGE can't get built because 16GB space is needed in /tmp while only 7.9 GB is available"
Relying on all packages to be correctly annotated by this won't be easy. BTW, firefox in particular also needs lots of RAM. On i686 it won't build anymore due to not fitting some processes into 32-bit address space.
@davidak Are you using boot.tmpOnTmpfs = true;
?
@worldofpeace yes.
I don't understand why the size of the tmpfs is 7.9G. I have 16 GB RAM.
And the systemd unit don't sets it:
[Mount]
What=tmpfs
Where=/tmp
Type=tmpfs
Options=mode=1777,strictatime,nosuid,nodev
(Get path with systemctl status tmp.mount
)
If you don't specify anything, tmpfs gets 50% of RAM. You can add e.g. size=12G
to options. (I use that live: mount -o remount,size=12G
.)
I feel like this now naturally drifts into having https://github.com/NixOS/nixpkgs/issues/23912
Relying on all packages to be correctly annotated by this won't be easy. BTW, firefox in particular also needs lots of RAM
@vcunat Yes, it will get outdated quickly. Maybe hydra can capture the usage before deleting the build directory and saves as statistics. Then we could update the packages automatically before every stable release.
I have disabled huge desktop programs like chromium, firefox and libreoffice for now, since i mostly want to test the pantheon desktop.
Yes, theoretically, but I don't think it will be very reliable anyway. If you run two builds at once, for example (not a rare thing IMO).
@vcunat but they don't use the same directory, right?
Just du
before deleting, should be quiet easy.
I installed NixOS on a thinkpad with 256 MB RAM once. It shouldn't start building firefox there. That would just be a waste of time and bad UX!
I meant the detection if there's enough free space before the build starts.
By the way, to avoid having to rebuild the world when you want to use something from staging, it is often possible to checkout certain evaluation that was already build by hydra (one that has low number queued/grey jobs):
https://hydra.nixos.org/jobset/nixpkgs/staging
For example I can try the first one with only 257 queued jobs and not many failures
I make sure the pantheon changes are there:
and then I try to build from that commit. Thankfully, none of the failures are affecting us so Nix will just fetch the build derivations.
@jtojnar oh, thanks! i didn't know that staging is also build since there is no channel.
Do you know the workflow with staging and staging-next? What is merged into what and when? (especially when will staging get merged in master which is hopefully less broken?) I didn't found that documented.
Do you know the workflow with staging and staging-next? I didn't found that documented.
Best documentation we have currently is the RFC.
Now there's also a habit opening a PR for each staging-next merge, so we get a thread to discuss stabilization for the particular iteration. Current one: https://github.com/NixOS/nixpkgs/pull/54306
@vcunat One can just add all the concurrent builds and compare that to the amount of space available. The error message then would be "you are trying to build package A(1GB) and B(16GB). Your available space is 8GB. This means that building is not possible. Please upgrade hardware or optimize the build.".
Why can't you just have some variable/setting to use a disk folder rather than tmpfs? It seems stupid to kill building in these situations, as if somehow using the latter wasn't simply out of habit.
The FS for /tmp
is choice of your system, and you can configure it any way you want. I believe nix
(typically daemon) respects the $TMPDIR
variable, so you can even separate that from the rest of the system.
Right, it would make sense to have a free space check in either scenario.
Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:
This is still an issue.
Just got it reproduced with 8GB RAM allocated for my VirtualBox builder.
This might be an instance of https://github.com/systemd/systemd/commit/7d85383edbab73274dc81cc888d884bb01070bc2#diff-ff9de8d0d78a6434e92fcd42eca4801d682efdc8c0539f901022b4374bd94a13R25, which got reverted in https://github.com/systemd/systemd/commit/362a55fc14a8933e0184fb79114fc91d54d1bb40.
Can you try cat /proc/mounts
after a fresh reboot?
Relying on all packages to be correctly annotated by this won't be easy. BTW, firefox in particular also needs lots of RAM. On i686 it won't build anymore due to not fitting some processes into 32-bit address space.
In the Gentoo portage tree, ebuilds with bigger storage requirements are annotated, so it is probably feasable... It is enough to do this on a best-effort basis, to improve error messages, it doesn't directly influence the build...
g++ -c -pipe -O2 -std=c++1z -fvisibility=hidden -fvisibility-inlines-hidden -fno-exceptions -Wall -Wextra -Wvla -Wdate-time -Wshift-overflow=2 -Wduplicated-cond -Wno-stringop-overflow -Wno-format-overflow -D_REENTRANT -fPIC -DQT_NO_LINKED_LIST -DQT_NO_JAVA_STYLE_ITERATORS -DQT_DEPRECATED_WARNINGS -DQT_NO_NARROWING_CONVERSIONS_IN_CONNECT -DQT_NO_EXCEPTIONS -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -DQT_NO_DEBUG -DQT_PLUGIN -DQT_NETWORK_LIB -DQT_DBUS_LIB -DQT_CORE_LIB -I. -I../../../../include/QtNetwork/5.15.2 -I../../../../include/QtNetwork/5.15.2/QtNetwork -I../../../../include/QtCore/5.15.2 -I../../../../include/QtCore/5.15.2/QtCore -I../../../../include -I../../../../include/QtNetwork -I../../../../include/QtDBus -I../../../../include/QtCore -I.moc -I/nix/store/am3dsgzyqm904n4l7jvf1laj5nmirf92-icu4c-68.2-dev/include -I/nix/store/dzh9ikjl7cybjbdn7jmdlx9kj466rqqp-zlib-1.2.11-dev/include -I/nix/store/3fqa135a3z30nkn8ibl2s517al8p5aiv-libjpeg-turbo-2.0.6-dev/include -I/nix/store/a8jjncyjbp99iskjhxksa0sdahyfjbqz-harfbuzz-2.8.0-dev/include -I/nix/store/3hn7ib1w1frmp1m062w9maw596fa6f6b-openssl-1.1.1k-dev/include -I/nix/store/8hgmgibwipsh2ds4gfgl2rrdhx7kgxmb-libX11-1.7.0/include -I/nix/store/hsvlpwb1bnx09x97rqzpk3n02j8psapz-libXext-1.3.4/include -I/nix/store/xmqvb6sv5l1d829g66kcrnxjqjy8hnb1-libXrender-0.9.10/include -I/nix/store/8sk035v2pxg6hjp9cx4dsd2wncnzns24-cups-2.3.3-dev/include -I/nix/store/37dwzcg8b95icnnx9vajdxm7m6s0ircc-mariadb-connector-c-3.1.8/include -I../../../../mkspecs/linux-g++ -o .obj/main.o main.cpp
CC [M] drivers/atm/uPD98402.o
CC [M] drivers/uio/uio_dmem_genirq.o
CC [M] drivers/vfio/pci/vfio_pci_rdwr.o
CC [M] drivers/staging/wfx/fwio.o
CC [M] drivers/pcmcia/pcmcia_resource.o
CC [M] drivers/block/aoe/aoedev.o
CC [M] drivers/uio/uio_aec.o
CC [M] drivers/atm/nicstar.o
CC [M] drivers/vfio/pci/vfio_pci_config.o
CC [M] drivers/uio/uio_sercos3.o
CC [M] drivers/pcmcia/cistpl.o
CC [M] drivers/staging/wfx/hif_tx_mib.o
CC [M] drivers/block/aoe/aoemain.o
CC [M] drivers/uio/uio_pci_generic.o
CC [M] drivers/vfio/pci/vfio_pci_igd.o
/nix/store/k934q7j0zpxl7311kgklb4jcsyijqjrn-binutils-2.35.1/bin/ld: final link failed: No space left on device
../../gcc-10.2.0/gcc/convert.c:1133:1: fatal error: error writing to /build/ccnbG7zw.s: No space left on device
1133 | }
| ^
compilation terminated.
make[3]: *** [Makefile:1117: convert.o] Error 1
make[3]: *** Waiting for unfinished jobs....
collect2: error: ld returned 1 exit status
make[2]: *** [tools/llvm-shlib/CMakeFiles/LLVM.dir/build.make:296: lib/libLLVM-7.1.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:24890: tools/llvm-shlib/CMakeFiles/LLVM.dir/all] Error 2
make: *** [Makefile:171: all] Error 2
error: builder for '/nix/store/j3s3pylmmvsz4pig84d246x38z8s7z73-llvm-7.1.0.drv' failed with exit code 2
error: 1 dependencies of derivation '/nix/store/k8rd99n5m9hl1fkwgxafs7i5qk2nfmlp-darktable-3.4.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/jd4w7lvssx74vlmvjx7shy2nqh0i8hhf-spidermonkey-68.10.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/4ggn71pm07zsi3qdmqrbmxdwzia2kq4i-libproxy-0.4.17.drv' failed to build
error: 1 dependencies of derivation '/nix/store/wh43jlpr9v9w87qil302m6gml8hk8psk-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/2ar92z06ir62q7fsxlq7hn1mn49ija1w-nixos-system-gaming-21.05pre289526.7a1fbc38a4b.drv' failed to build
llvm needs more than 15 GB
One idea would be to have hydra calculate the size of the build folder just before deleting it and have it store it somewhere
This would be useful to quickly aggregate build space required for a lot of packages and the big ones could be annotated.
Doing it without changing the derivation should be possible with a patch to nix or hydra.
Then another patch for nix, possibly through the existing features system. For example "16gb-build-space" or similar. This could be dynamically generated for the local host based on the actual space on /tmp free
For a single-user nixpgs installation, I think you can just set TMPDIR
when you build something you know is going to be big, eg TMPDIR=/var/tmp nix-build ...
. However, for a multi-user installation (eg NixOS) you have to set the env var in nix-daemon
. The simplest way I've found to do that is to set nix.envVars.TMPDIR = "/var/tmp";
in /etc/nixos/configuration.nix
. However, that overrides TMPDIR
for the entire system, not just for nix-daemon
, and I don't want that.
I would like to have a way to set env vars specifically for nix-daemon
. Perhaps it could be done using pre-build-hook
in /etc/nix/nix.conf
but currently that doesn't support setting env vars, only adding extra-sandbox-paths
. Another alternative might be to use an overlay that modifies systemd.services.nix-daemon
. However, neither of these is at all intuitive and I would much prefer to be able to use nix.envVars
and not have them be added to environment.sessionVariables
, or to have an additional config option like nix.daemon.envVars
.
Here is another fun aspect of this: When building a NixOS config using ZFS a 2GB Raspberry PI 4, I kept getting an error like this, no matter how much space (swap) I allowed for the tmpfs (/run/user/0
).
Turns out I wasn't running out of space, but rather out of inodes. The default size for /run/user/*
is about 10% of RAM, and an inode limit that may (or may not) make sense for that size.
Changing the size of the tmpfs (mount -o remount,size=XG /run/user/0
) does not change that inode limit.
An additional nr_inodes=0
removes the limit and fixes the problem.
https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt lists some concerns for setting nr_inodes=0
; I am not sure whether those also apply to a file system reserved for root
.
If not, it may be best (at least for nix on NixOS) to just set nr_inodes=0
for /run/user/0
by default.
Otherwise, any solution to avoid (or warn about) running out of space should also consider running out of inodes (in case the target file system still uses a finite number of them).
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
Wow, I've been using linux and NixOS for some time... but this issue really cost me a lot of time. Maybe if the message were:
note: build failure may have been caused by lack of free disk space (A common cause is the default tmpOnTmpfs size is too small)
rather than just:
note: build failure may have been caused by lack of free disk space
that wouldn't have been the case.
As a workaround for being unable to change $TMPDIR
for a running nix-daemon (i.e. you use NixOS), I wrote a little script for myself which increases /tmp
size by adding swap (on a btrfs rootfs):
readonly SWAPFILE="/swap/swapfile"
[ ! -d /swap ] && sudo btrfs subvolume create /swap
if [ -f "$SWAPFILE" ]; then
echo "error, $SWAPFILE already exists"
exit 2
fi
sudo touch "$SWAPFILE"
sudo chmod 600 "$SWAPFILE"
sudo chattr -c "$SWAPFILE"
sudo chattr +C "$SWAPFILE"
sudo fallocate -l 32G "$SWAPFILE"
sudo mkswap "$SWAPFILE"
sudo swapon "$SWAPFILE"
sudo mount -oremount,size=32G /tmp
It was really useful when building linux kernel several times (while 16 GiB RAM for /tmp
is not enough for storing its temporary build products).
I'm using boot.tmpOnTmpfs = true
, but have a sizeable ZFS dataset for /nix
I'd rather have builds take place in.
I've configured nix-daemon
accordingly:
{
systemd.services.nix-daemon = {
environment.TMPDIR = "/nix/tmp";
};
systemd.tmpfiles.rules = [
"d /nix/tmp 0755 root root 1d"
];
}
This seems to run builds in the right filesystem:
edef@jaguar ~> stat -c %d /tmp
53
edef@jaguar ~> stat -c %d /nix/tmp
30
edef@jaguar ~> cat (nix-build -E '(import <nixpkgs> {}).runCommand "stat" {} "stat -c %d . > $out"' 2>/dev/null)
30
Hopefully this ensures I can run parallel builds without them chewing through 32G of tmpfs (or ~64M inodes) and dying of ENOSPC
.
Configuring nix-daemon
with a separate TMPDIR
is smart - I still wonder if there's any particular reason for Nix to not just use /var/tmp
, like portage
also does.
Ensuring that /nix/store
shares a filesystem with builds may be helpful for making things like reflink and copy_file_range
work for speeding up certain builds a bit. It feels reasonable to keep Nix contained to /nix
, anyhow.
In my specific case, /nix
is its own ZFS dataset. @grahamc does likewise, which is likely to serve as inspiration for others. ZFS doesn't currently implement any significant optimisations for same-dataset copies, but those are being actively worked on.
there is a use case against "build nix stuff completely in /nix/store
" (it already builds the chroot directory there to be able to hardlink single-file store contents): /nix
on SSDs. Some builds chew through massive amounts of memory, and produce only comparatively tiny outputs (some browsers and such come to mind), but building them on an SSD seems unwise. And afaik the build outputs already get put into /nix/store
directly (as-in they get put into the chroot directory, then moved or such idk). But I'm also ok with the current state as-is (but I have 24GiB RAM + 24 GiB swap)...
mount -o remount,size=8G /tmp
worked for me, but the size resets back to 2G after reboot. Is there a way to do this permanently?
@MikiVanousek manually adding /tmp
to fileSystems
in configuration.nix
maybe, and specifying the size explicitly as an option, e.g.
fileSystems."/tmp" = {
device = "tmpfs";
fsType = "tmpfs";
options = ["noatime" "nodev" "size=8G"];
};
Use boot.tmpOnTmpfsSize in configuration.nix
@neilmayhew
Use boot.tmpOnTmpfsSize in configuration.nix
That is a percentage, right? Setting that to 100% would mean what size? I tried it and /tmp appears to be 1.9G.
@zseri
@MikiVanousek manually adding
/tmp
tofileSystems
inconfiguration.nix
maybe, and specifying the size explicitly as an option, e.g.fileSystems."/tmp" = { device = "tmpfs"; fsType = "tmpfs"; options = ["noatime" "nodev" "size=8G"]; };
I put that in my configuration, but after reboot the size is again 1.9G:
Filesystem Size Used Avail Use% Mounted on devtmpfs 190M 0 190M 0% /dev tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs 947M 4.1M 943M 1% /run tmpfs 1.9G 384K 1.9G 1% /run/wrappers /dev/sda2 453G 78G 353G 19% / tmpfs 1.9G 0 1.9G 0% /tmp /dev/sda1 500M 369M 131M 74% /boot tmpfs 379M 0 379M 0% /run/user/1000
@MikiVanousek what's the content of /etc/fstab
and the output of mount
? Also boot.tmpOnTmpfs
, boot.tmpOnTmpfsSize
override this via a different mechanism and will probably conflict if both fileSystems."/tmp"
and boot.tmpOnTmpfs
is used.
Output of ls /etc/systemd/system/*.mount
would also be useful.
See man tmpfs for the meaning of size.
@neilmayhew I have the following in my /etc/fstab
:
# Filesystems.
tmpfs / tmpfs size=30G,noatime,nodev,mode=755 0 0
and it works as expected.
Issue description
I'm trying to rebuild with staging branch.
But the rebuild fails:
Steps to reproduce
boot.tmpOnTmpfs = true;
rebuild on staging, building all the things
Workaround
mount -o remount,size=15G /tmp
Technical details
"x86_64-linux"
Linux 4.19.16, NixOS, 19.03.git.8291376 (Koi)
yes
yes
nix-env (Nix) 2.2
"nixos-18.09.1829.0396345b794, nixos-hardware, nixos-unstable-19.03pre164715.201d739b0ff"
/nix/var/nix/profiles/per-user/root/channels/nixos