Open copumpkin opened 9 years ago
Does nix-store -l /nix/store/slv1g031dm2f4wbd35zd9fkw8yay4ps7-CF-855.17.drv
show anything?
Next time I encounter it I'll let you know (also I haven't tracked it carefully enough to see if it's always CF
failing). This last time I just went and resumed the build.
Just got it again: the failed build wasn't CF this time, but was groff. In this case, the end of the log from nix-store -l
said:
clang++ -I. -I. -I/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/include -I/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/
building itoa.o
clang -I. -I. -I/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/include -I/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/in
make[2]: *** [itoa.o] Killed: 9
make[2]: Leaving directory `/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/libs/libgroff'
make[1]: *** [src/libs/libgroff] Error 2
make[1]: Leaving directory `/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3'
make: *** [all] Error 2
The general log said:
note: keeping build directory ‘/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1’
builder for ‘/nix/store/d7dv8wckdxv263v20d6pnb11p98h89vh-groff-1.22.3.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/xbf62r96p0kgmn8y654bc980cayi77n4-libedit-20140620-3.1.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/zgxp3ahihp1v463mn87mpaf080cqa1bp-llvm-3.5.0.drv’: 1 dependencies couldn't be built
killing process 89785
note: keeping build directory ‘/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-coreutils-8.23.drv-0’
killing process 89788
killing process 89788: Operation not permitted
note: keeping build directory ‘/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-openssl-1.0.1k.drv-0’
killing process 89784
note: keeping build directory ‘/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-bison-3.0.4.drv-0’
cannot build derivation ‘/nix/store/xq0jzq1a7x6xh3gn6jrvzp5vh1h9zxk4-stdenv-darwin.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/xq0jzq1a7x6xh3gn6jrvzp5vh1h9zxk4-stdenv-darwin.drv’ failed
So something literally sent the build a kill signal. I looked in my syslog for evidence of sandbox stuff killing it, but couldn't find anything. I'm assuming the operation not permitted thing might be because the process is already dead, due to the other signal?
Yeah, I think the "operation not permitted" is a red herring here, the Killed: 9 is the kicker. Could it be OOM or something?
OOM seems unlikely; beefy box. Can't help feeling it has something to do with Mac OS sandboxing, but I see no evidence of that. For the record, I'm running with --cores 8 -j 8
, and am not sure how common that is on linux builds.
Hydra runs with some huge numbers there
Looking more closely, I think this is a Mac OS bug. It just happened again and I noticed crash reports for the same Mac system process (taskgated
) highly correlated with the times of my posts here. I'm guessing the sandboxing mechanism uses this daemon, somehow crashes it, and when it crashes the daemon, the sandboxed process gets killed. Probably safe to close this ticket unless you want me to post updates as I learn them.
I marked this as stale due to inactivity. → More info
I closed this issue due to inactivity. → More info
I'm pretty sure all the things being built are
make -jN
-safe, so I think this is either a nix problem or a problem with my Mac OS sandboxing code:Rerunning the exact same command again then proceeds to complete successfully.
Really painful to track down when they happen so infrequently and unreliably. I'll probably look into them more carefully once I get a computer that can handle this sort of parallelism easily.