search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.57k stars 1.5k forks source link

Intermittent "operation not permitted" errors when doing very parallel builds #471

Open copumpkin opened 9 years ago

copumpkin commented 9 years ago

I'm pretty sure all the things being built are make -jN-safe, so I think this is either a nix problem or a problem with my Mac OS sandboxing code:

$ nix-build -Q -j8 --cores 8 --option build-use-chroot true --option use-binary-caches false -A stdenv
these derivations will be built:
  /nix/store/y5d9cxdpgyp4ccf66mfg3s22mr06fxwd-clang-wrapper-9.9.9.drv
  /nix/store/wxbbc1pxv0gqdhrfy00plq46mpk2qy6v-stdenv-darwin-boot-1.drv
  /nix/store/vn6aziw2d9614s4b3w0xyj44s0j9cf1k-gnum4-1.4.17.drv
  /nix/store/0b79qm21360f4rq9bn8yslxm9flhql1y-gmp-5.1.3.drv
  /nix/store/4lbps6zbn3c95j2grbqw3vfli4s2nv70-bash43-033.drv
  /nix/store/4r6pwnynmq8vwbvrb8c9vxj15b01zhc9-bash43-031.drv
  /nix/store/0jdyzc0kaq9v6qfhf6q8gi0cjn67m02i-dyld-239.4.drv
  /nix/store/3ymq5vc6x9y3fnszxd25qf9q7w4pcwiq-Libm-2026.drv
  /nix/store/522vhinhivzlgmcn4dn1yfrvcqz4nnjb-cpio-2.11.drv
  /nix/store/5g3hjfg7l9x09iyanb3q8pjgm3w7s0f5-Csu-79.drv
  /nix/store/5kqmkmhqd3v26vrfn4jrzj1wbpvxsqgl-ncurses-5.9.drv
  /nix/store/bymmy3fq9zbb3hjd4fzqik8wcflkk15d-ed-1.10.drv
  /nix/store/vq3dvnnmbwjzqd6nasijg6lfnzgj37vj-unifdef-2.6.drv
  /nix/store/8jybavgyw7zji7qb53076k31dfavi7vk-Libc-825.40.1.drv
  /nix/store/6sakx1y1i9xjjd6jq5yvr036pz56ankn-Libc-997.90.3.drv
  /nix/store/76q07lb07d9x896m1j5lkf5i7z95nbc3-Libnotify-121.20.1.drv
  /nix/store/8jij6fzbfgw399c5rwqa33296iax440i-perl-5.20.1.drv
  /nix/store/kd3s9bw5a40cqzw6li2khvrl9jlq2y23-bison-3.0.2.drv
  /nix/store/r073y51mlrpk8ni43b6gd61c4jccgwa8-flex-2.5.39.drv
  /nix/store/451v9yyrmax0k1y7bp08rq4sr4vnx2kp-bootstrap_cmds-86.drv
  /nix/store/s891bn7xhnj8sgp9w2jibmcm98ikba7v-xnu-2422.115.4.drv
  /nix/store/wr8gngafm1pnfdg0b3mwyvgb411qgksq-libdispatch-339.92.1.drv
  /nix/store/8fi168qdgx416y0l0xp9b78cn7r3rs5l-libpthread-105.1.4.drv
  /nix/store/agwc1p4ssm2zghs790d5r97rjqhx63vl-Libinfo-449.1.3.drv
  /nix/store/f0danpm0vfgh05wqilcrr97w65b2bly5-CarbonHeaders-9A581.drv
  /nix/store/hdr78khfhavdy7ws6x6pwlvni7nmbg6s-libclosure-63.drv
  /nix/store/i2rr0qb3adczdr7iiz9akgbdz4nik340-copyfile-103.92.1.drv
  /nix/store/ic9f8lr5z14aja9dclxin8lzg744gha6-architecture-265.drv
  /nix/store/kcg5i4zbrf0dc3gib3mvv6zg5mx5a9sy-removefile-33.drv
  /nix/store/qm0nqz26w8p1h5b7s5a874fdk1s2k4lg-mDNSResponder-522.92.1.drv
  /nix/store/97yiyhd2d2345p3d7f9iyb5c3420cvx0-eap8021x-180.drv
  /nix/store/achzgxqdriq0hvnsw3pfb514hka6c0b5-IOKit-907.100.13.drv
  /nix/store/rjs9c7aqba62mapl9qw3zax565b0gygy-ppp-727.90.1.drv
  /nix/store/zgspi50bvaihaiqcjs93sn48v390w90q-Security-55471.14.18.drv
  /nix/store/zl6x93vz6y6jb6x6aabm8sll7a6s4lql-launchd-842.92.1.drv
  /nix/store/sjm4lzhp6v79h6fs2sd6l15qv0zhamcb-configd-453.19.drv
  /nix/store/sv2xx5g2zsnwiwawlpbd7rbjs8d9y4qn-libresolv-54.drv
  /nix/store/vbzn2ckpvdzwy4xpg8ifrkbabvn89rih-autoconf-2.69.drv
  /nix/store/ys3qvbxg8vyfnas1cvvg2mgxfsjawfhq-hook.drv
  /nix/store/71ngvzgbrg03nxkrr5w4zka978rjn2ml-automake-1.12.6.drv
  /nix/store/kgbvq3bf1874wapp9ysz3fikwvxaig8j-objc4-551.1.drv
  /nix/store/scxqnz25bihzm2jsl387p93xibmbciqw-openssl-1.0.1k.drv
  /nix/store/yfg95d7ca6m9c85m0agij02pajqbs02q-libtool-2.4.2.drv
  /nix/store/vwdh8qmwms2dljy36bq2djhpdw3wy472-cctools-port-862.drv
  /nix/store/zq7kh9c7qv3b176v50p5y0sp0bbbz41z-CommonCrypto-60049.drv
  /nix/store/4177xiw1vdcln5vjlixbhfy60i2xvvxy-Libsystem-1197.1.1.drv
  /nix/store/643dmpcg4ya91ijg9df1xfxp5hgb3wyh-hook.drv
  /nix/store/6d3l4l1xn8yycllfffryys4y9axdp35a-icu4c-53.1.drv
  /nix/store/4jprnl5fpdf4iwdq6bja4m4yrl51x8gz-libiconv-1.14.drv
  /nix/store/mz0smaw8a8q2n0yzyg85rcbd77ypngh1-gettext-0.18.2.drv
  /nix/store/wsl4a8wphfp8xam18kqkdmzmnjy7a41m-coreutils-8.23.drv
  /nix/store/5xyqkhdi9f8g6wfx9w07rsc3y4ml5124-sharutils-4.11.1.drv
  /nix/store/cf2sf5z32dy50j26xb7drbvfd7gpwp95-zlib-1.2.8.drv
  /nix/store/h0p36nn7rnml6lrmrkgmimzvvmcgg2m9-hook.drv
  /nix/store/80kiynxbj16y2w986y7gnbp6y9n9xv3i-libxml2-2.9.2.drv
  /nix/store/kpkd77y04k4lnn077psbs1sxcxwybhz2-bzip2-1.0.6.drv
  /nix/store/0ynss9ql2jk3cfayhz2mmg8zgywzgd3b-libarchive-3.1.2.drv
  /nix/store/65jbhf72zxxdpaalg3j1fc8f79bsbvyr-expat-2.1.0.drv
  /nix/store/dpvylnvg7f1q2z4rjs7nqf7i6364jjf9-libssh2-1.4.3.drv
  /nix/store/a8z1l0sn51m32hsrpd12cy85d4jkvxsf-curl-7.39.0.drv
  /nix/store/fmj7xk62cidwdd44gi8rbm6i93zdl01f-ps-153.drv
  /nix/store/ip3200b5hb370hc0dcx69rl7l5rwafl3-cmake-2.8.12.2.drv
  /nix/store/w5ppn91qfsn0hg86jfijjaqyaiqknqdd-libcxxabi-pre-217324.drv
  /nix/store/gahrq5ilflr20x1jfhhcy52vpxi94alq-libc++-3.5.0.drv
  /nix/store/q63scj4vlw2kzyvfjcpydm1309ngr6jy-clang-wrapper-9.9.9.drv
  /nix/store/slv1g031dm2f4wbd35zd9fkw8yay4ps7-CF-855.17.drv
  /nix/store/6li3caiqhxp1hssl6w04p46vxscap8kp-stdenv-darwin-boot-2.drv
  /nix/store/dy8f4aj627yb4mj5w7s4nisy9vr985pc-bash43-032.drv
  /nix/store/97xl5ih3rxx3w9hni5284842p5ywz298-bash-4.3-p33.drv
  /nix/store/337ymlamir4zjivzl63nfs5zpgc71nam-CF-855.17.drv
  /nix/store/5h8xpw7l54vvy09g6amba9ga9ijlkpd2-libcxxabi-pre-217324.drv
  /nix/store/8w9cf340a4y0ind8i3dwm316vab3iggp-clang-wrapper-9.9.9.drv
  /nix/store/jhyq3v1gycnwz0hc8bnkqr5kq728dc3k-hook.drv
  /nix/store/zmj3sxbchiw9dq6937hyj4jbm693vd3k-libc++-3.5.0.drv
  /nix/store/mddr6hf5ssnb38ygri61pc9j85b0vfhv-stdenv-darwin-boot-3.drv
  /nix/store/8g9x65w7j08s6lgj7srgj8xqvfhd9j22-zlib-1.2.8.drv
  /nix/store/i5yfmgd4zvn84900amagcjji7z5y16q7-python-2.7.9.drv
  /nix/store/kjb06q3gsi36hhcnlr3kpq7pl95kwqi8-groff-1.22.3.drv
  /nix/store/law7d2x2k93mjlzs2gkd904aklvxx6af-ncurses-5.9.drv
  /nix/store/pg23cjmczjrd77xafd57yvcga60j76y2-hook.drv
  /nix/store/mpkic7wa8yh739vzgi64qgfrbmkfsd1v-libxml2-2.9.2.drv
  /nix/store/ch2a36s6n6f60v18qj4hyazyjn7962js-objc4-551.1.drv
  /nix/store/nvh5bvg36x2sbr53h3hg17hpqdbnxnrn-cctools-port-862.drv
  /nix/store/r76d4mgiph16fh4kwpdydxc8qz7vrpwr-libffi-3.0.13.drv
  /nix/store/0cqxm72dcn5glinhc9jr68vzp75xg895-llvm-3.5.0.drv
  /nix/store/srw4w96vqwpddwig95a2ddl7hfr3rbn8-xz-5.2.0.tar.bz2.drv
  /nix/store/6nzwkr5sn90yi4vgpzp2sc3j7c9c2sdd-xz-5.2.0.drv
  /nix/store/81i36kyhk2ancrqizqbgkmrin1ybsdif-pcre-8.36.drv
  /nix/store/pnfzkjhrx5r0zwdfgppmwf33ry3kj2hj-hook.drv
  /nix/store/8csmkbhi5xvq3dxr5sk22pxhmd6kn1rc-icu4c-53.1.drv
  /nix/store/xddnfnm2xk9gwnfmkqjsxx40xap1km56-libedit-20140620-3.1.drv
  /nix/store/aiaprc69221836pkmp9glr23cgyamfqp-clang-3.5.0.drv
  /nix/store/v3qzpj8fd09ncpzdxcvsmq9xa20bx9j2-gmp-5.1.3.drv
  /nix/store/sq1ns53f03ji87yisb4krg7sxc08b8dv-coreutils-8.23.drv
  /nix/store/by1h9y682b7m6v3zsprc07xmpwlqnv8a-diffutils-3.3.drv
  /nix/store/i09vb0q9v4n9fs1x4isxjpnvcayl6n28-libiconv-1.14.drv
  /nix/store/cq4di52yxlxgr5cvhakhdd04jr83brdd-gnugrep-2.20.drv
  /nix/store/fbgdwlkmv9c3pv3ppzsarh92wji78fs8-gnumake-3.82.drv
  /nix/store/fc8rrlijyfc8r9iiz73r9mg3xgpw18w1-gzip-1.6.drv
  /nix/store/i0034qi3l754lc4rvg6yd86c39xb8w7i-ed-1.10.drv
  /nix/store/iqxfi6lr1n6ng2wj79mgpfbf69lj4yzz-bzip2-1.0.6.drv
  /nix/store/j1bszcnvjfcdky0nh6y73np34q6hii4y-clang-wrapper-3.5.0.drv
  /nix/store/w0ynyv9abas8dl06pcapn2acq00rs1kc-libsigsegv-2.10.drv
  /nix/store/jfizqmp421rijp2rb37bwzkb9gdwfq3w-gawk-4.1.0.drv
  /nix/store/khzr3fr73hrii5jdm437g7jl8mma00hw-gnutar-1.27.1.drv
  /nix/store/kvgqx74ndlxdnwy4zpmacpz8n4hrhc9f-findutils-4.4.2.drv
  /nix/store/m3brb8rkcydj4ld7k34mcc7nazkw2k45-CF-855.17.drv
  /nix/store/mfpmz87695k6l09y82b6bnznbfnj4wfj-patch-2.7.1.drv
  /nix/store/p4yadwwwgjjkq0ibb4jcl2cjf8fp11lj-gnused-4.2.2.drv
  /nix/store/qfqj2a2zg98jfg8azdwb9cv3miskrm61-stdenv-darwin.drv
building path(s) ‘/nix/store/k728f9bxb6mkn1qkxf3f1yl37iqypdk9-bash43-031’
building path(s) ‘/nix/store/dkzh0n5wr5mk3hryl1i79ncwas0s3qrk-bash43-032’
building path(s) ‘/nix/store/znd6apsyigf80vzyxg5651ip345fzxay-bash43-033’
building path(s) ‘/nix/store/g13iiyh5mqsz6qzf26j1s7wil8hs5mqj-xz-5.2.0.tar.bz2’
building path(s) ‘/nix/store/7aajk7gnf5r5wnyh40xjyxa1sa6gq23x-clang-wrapper-9.9.9’
building path(s) ‘/nix/store/hfgc9vvg701fb49y12q7ryqjw2vj78gy-stdenv-darwin-boot-1’
building path(s) ‘/nix/store/pnf941v06f6454pf3b55lqxhysdppwff-CarbonHeaders-9A581’
building path(s) ‘/nix/store/2wj03mjdv54jh4lcdj0gnbdhs5kpagc8-CommonCrypto-60049’
building path(s) ‘/nix/store/14bkyv9rbw23c2r1776yc8mkm40gh471-Csu-79’
building path(s) ‘/nix/store/wm6avhf68iyp7jfbq2qqj7zc60w8z3cp-Libinfo-449.1.3’
building path(s) ‘/nix/store/93hdfjs38b4pszg1kslbamnwhpza0mzm-Libm-2026’
building path(s) ‘/nix/store/5nnx0cni07d4d3zl0rm29y5kc83kdpwc-Libnotify-121.20.1’
building path(s) ‘/nix/store/dc42h7ggbary27ypfab7r3b02azvcd0m-Security-55471.14.18’
building path(s) ‘/nix/store/6wkizb3p5ya9iczk88n7zxwjnbpgq8ka-architecture-265’
building path(s) ‘/nix/store/gi3f0pjk90m01ip2x495gh82rgvxks0q-bzip2-1.0.6’
building path(s) ‘/nix/store/wr3wb98lvshsnmdrr2schxs5ksxqb5la-copyfile-103.92.1’
building path(s) ‘/nix/store/v0sqjmfb00rlc0akk5pi98v0ldzim6x4-cpio-2.11’
building path(s) ‘/nix/store/fmg08k55pr6d24g8yvg75xh100yxghjc-dyld-239.4’
building path(s) ‘/nix/store/ixcj7bxbpy79165z7ydxhyiwhbp8gs9z-eap8021x-180’
building path(s) ‘/nix/store/d1gm5rlmj89lnk28prgmg1dkv7zzddjg-ed-1.10’
building path(s) ‘/nix/store/5g3aiq4vi2n5nl6khgzw0mna72lmqgxa-expat-2.1.0’
building path(s) ‘/nix/store/cz7hlxzqgyqbsn292irnhn17gm49bma5-gnum4-1.4.17’
building path(s) ‘/nix/store/h3vmxbmj10j7dkynnr0ckna5srsx4mgc-hook’
building path(s) ‘/nix/store/d87a4jxw2v0741kvflgjz720wzj961cp-hook’
building path(s) ‘/nix/store/ks8fh4mg05dpjwxzqv05d3wwibmp68vp-hook’
building path(s) ‘/nix/store/8ri93wpxcbpgm5wqhba9ij5mb7z4k85m-icu4c-53.1’
building path(s) ‘/nix/store/a2jgampqgg9dz6rg7jh85fv2w3ax41l5-launchd-842.92.1’
building path(s) ‘/nix/store/ljrisaazkmb0s5j69s3wn0cax0ir61rf-libclosure-63’
building path(s) ‘/nix/store/i8jm50w0wkb8nc6d60sh9m00daa186p3-libdispatch-339.92.1’
building path(s) ‘/nix/store/d4fhvjj7n0fzda19lkikimf0xhvp4aam-libiconv-1.14’
building path(s) ‘/nix/store/vn77hffdx9za9a9d1960xwgpaz1j8836-mDNSResponder-522.92.1’
building path(s) ‘/nix/store/njg4q4f2gz5ivyfpvl80ljaz47b649m8-ncurses-5.9’
building path(s) ‘/nix/store/zq0paa8gfn3sc08iz9kpsmgjggw3g3qy-objc4-551.1’
building path(s) ‘/nix/store/cph9vlqg2fbklpphawpra8i1skmm64rs-perl-5.20.1’
building path(s) ‘/nix/store/0wb9jdsg91w5hzhyca7c7w6jq3w888zf-ppp-727.90.1’
building path(s) ‘/nix/store/45q0jp2vcvdjq87cbnzyh6rnm2vx8dn9-ps-153’
building path(s) ‘/nix/store/7vaa345qn50mxq7csm9h8y4j1divp780-removefile-33’
building path(s) ‘/nix/store/bnblrlk89v17p0yik55j8a5s3q3vs02r-unifdef-2.6’
building path(s) ‘/nix/store/hbqa8vizpkad3hzxr48arrlxxxzsf309-zlib-1.2.8’
building path(s) ‘/nix/store/785zijqbrf9h1gmbzcapn5bm61hdsp2m-Libc-825.40.1’
building path(s) ‘/nix/store/z1szahw1gj5q9wgipngrf0vi28s8cwic-libxml2-2.9.2’
building path(s) ‘/nix/store/y0lqxjfrxkmfngc5r7dp2mznch8njbs1-Libc-997.90.3’
building path(s) ‘/nix/store/xd5v9hyznjb2n202ba7034ai06b9cwb9-gettext-0.18.2’
building path(s) ‘/nix/store/6xml2g8nn12f8j9b28bqidm8lahdb9fv-gmp-5.1.3’
building path(s) ‘/nix/store/wkqsqxhgkv5mznayw1j5nlm9hgjdd251-CF-855.17’
builder for ‘/nix/store/slv1g031dm2f4wbd35zd9fkw8yay4ps7-CF-855.17.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/i5yfmgd4zvn84900amagcjji7z5y16q7-python-2.7.9.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/6li3caiqhxp1hssl6w04p46vxscap8kp-stdenv-darwin-boot-2.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/337ymlamir4zjivzl63nfs5zpgc71nam-CF-855.17.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/97xl5ih3rxx3w9hni5284842p5ywz298-bash-4.3-p33.drv’: 1 dependencies couldn't be built
killing process 63703
killing process 59123
killing process 7228
killing process 7228: Operation not permitted

Rerunning the exact same command again then proceeds to complete successfully.

Really painful to track down when they happen so infrequently and unreliably. I'll probably look into them more carefully once I get a computer that can handle this sort of parallelism easily.

edolstra commented 9 years ago

Does nix-store -l /nix/store/slv1g031dm2f4wbd35zd9fkw8yay4ps7-CF-855.17.drv show anything?

copumpkin commented 9 years ago

Next time I encounter it I'll let you know (also I haven't tracked it carefully enough to see if it's always CF failing). This last time I just went and resumed the build.

copumpkin commented 9 years ago

Just got it again: the failed build wasn't CF this time, but was groff. In this case, the end of the log from nix-store -l said:

clang++ -I. -I. -I/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/include -I/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/
building itoa.o
clang -I. -I. -I/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/include -I/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/in
make[2]: *** [itoa.o] Killed: 9
make[2]: Leaving directory `/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3/src/libs/libgroff'
make[1]: *** [src/libs/libgroff] Error 2
make[1]: Leaving directory `/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1/groff-1.22.3'
make: *** [all] Error 2

The general log said:

note: keeping build directory ‘/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-groff-1.22.3.drv-1’
builder for ‘/nix/store/d7dv8wckdxv263v20d6pnb11p98h89vh-groff-1.22.3.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/xbf62r96p0kgmn8y654bc980cayi77n4-libedit-20140620-3.1.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/zgxp3ahihp1v463mn87mpaf080cqa1bp-llvm-3.5.0.drv’: 1 dependencies couldn't be built
killing process 89785
note: keeping build directory ‘/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-coreutils-8.23.drv-0’
killing process 89788
killing process 89788: Operation not permitted
note: keeping build directory ‘/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-openssl-1.0.1k.drv-0’
killing process 89784
note: keeping build directory ‘/private/var/folders/dz/l81blgwx25sf0yp4lz1v3rkr0000gq/T/nix-build-bison-3.0.4.drv-0’
cannot build derivation ‘/nix/store/xq0jzq1a7x6xh3gn6jrvzp5vh1h9zxk4-stdenv-darwin.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/xq0jzq1a7x6xh3gn6jrvzp5vh1h9zxk4-stdenv-darwin.drv’ failed

So something literally sent the build a kill signal. I looked in my syslog for evidence of sandbox stuff killing it, but couldn't find anything. I'm assuming the operation not permitted thing might be because the process is already dead, due to the other signal?

shlevy commented 9 years ago

Yeah, I think the "operation not permitted" is a red herring here, the Killed: 9 is the kicker. Could it be OOM or something?

copumpkin commented 9 years ago

OOM seems unlikely; beefy box. Can't help feeling it has something to do with Mac OS sandboxing, but I see no evidence of that. For the record, I'm running with --cores 8 -j 8, and am not sure how common that is on linux builds.

shlevy commented 9 years ago

Hydra runs with some huge numbers there

copumpkin commented 9 years ago

Looking more closely, I think this is a Mac OS bug. It just happened again and I noticed crash reports for the same Mac system process (taskgated) highly correlated with the times of my posts here. I'm guessing the sandboxing mechanism uses this daemon, somehow crashes it, and when it crashes the daemon, the sandboxed process gets killed. Probably safe to close this ticket unless you want me to post updates as I learn them.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

stale[bot] commented 2 years ago

I closed this issue due to inactivity. → More info