nix no longer tells me why I can't build remotely

[pikajude]

Derivation in question is a fetchgit derivation. In nix 1.11, when a remote builder refused to build a derivation, or nix didn't bother asking the remote builder, there would be a debug message along the lines of hook reply is "decline". No such message is printed anymore, making it a lot harder to figure out what's going wrong:

$ nix-store --realise /nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv -vvv
querying info about missing paths...
starting pool of 8 threads
starting substituter program ‘/nix/store/4j9jacx8mjd4jlj53wvymyhxq7dqyj5d-nix-1.11.14/libexec/nix/substituters/download-using-manifests.pl’
starting substituter program ‘/nix/store/4j9jacx8mjd4jlj53wvymyhxq7dqyj5d-nix-1.11.14/libexec/nix/substituters/download-from-binary-cache.pl’
reaping 4 worker threads
these derivations will be built:
  /nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv
building ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’
|   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: created
|   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: woken up
|   entered goal loop
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: init
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: loading derivation
|   |   acquiring global GC lock ‘/nix/var/nix/gc.lock’
|   |   acquiring read lock on ‘/nix/var/nix/temproots/87548’
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: have derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’: created
|   |   substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’: woken up
|   |   substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’: init
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’: trying next substituter
|   |   substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’: trying next substituter
|   |   substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’: trying next substituter
|   |   path ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’ is required, but there is no substituter that can build it
|   |   substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’: done
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: waitee ‘substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’’ done; 0 left
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: woken up
|   |   substitution of ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’: goal destroyed
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: all outputs substituted (maybe)
|   |   building of ‘/nix/store/bzvhbc586fayrp5savz2pgbm0iyp18x1-stdenv.drv’: created
|   |   building of ‘/nix/store/bzvhbc586fayrp5savz2pgbm0iyp18x1-stdenv.drv’: woken up
|   |   building of ‘/nix/store/knq494gr8l2ldq2l1n5hs227h8dsmq37-nss-cacert-3.31.drv’: created
|   |   building of ‘/nix/store/knq494gr8l2ldq2l1n5hs227h8dsmq37-nss-cacert-3.31.drv’: woken up
|   |   building of ‘/nix/store/ys1gdpj1y91cg8vf4ph7a3bx48dsphpg-git-2.14.1.drv’: created
|   |   building of ‘/nix/store/ys1gdpj1y91cg8vf4ph7a3bx48dsphpg-git-2.14.1.drv’: woken up
|   |   building of ‘/nix/store/zjf6ppcfmlvrbqqq6zklsga12ll3scmz-bash-4.4-p12.drv’: created
|   |   building of ‘/nix/store/zjf6ppcfmlvrbqqq6zklsga12ll3scmz-bash-4.4-p12.drv’: woken up
|   |   building of ‘/nix/store/zjf6ppcfmlvrbqqq6zklsga12ll3scmz-bash-4.4-p12.drv’: init
|   |   building of ‘/nix/store/zjf6ppcfmlvrbqqq6zklsga12ll3scmz-bash-4.4-p12.drv’: loading derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/zjf6ppcfmlvrbqqq6zklsga12ll3scmz-bash-4.4-p12.drv’: have derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/zjf6ppcfmlvrbqqq6zklsga12ll3scmz-bash-4.4-p12.drv’: done
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: waitee ‘building of ‘/nix/store/zjf6ppcfmlvrbqqq6zklsga12ll3scmz-bash-4.4-p12.drv’’ done; 3 left
|   |   building of ‘/nix/store/ys1gdpj1y91cg8vf4ph7a3bx48dsphpg-git-2.14.1.drv’: init
|   |   building of ‘/nix/store/ys1gdpj1y91cg8vf4ph7a3bx48dsphpg-git-2.14.1.drv’: loading derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/ys1gdpj1y91cg8vf4ph7a3bx48dsphpg-git-2.14.1.drv’: have derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/ys1gdpj1y91cg8vf4ph7a3bx48dsphpg-git-2.14.1.drv’: done
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: waitee ‘building of ‘/nix/store/ys1gdpj1y91cg8vf4ph7a3bx48dsphpg-git-2.14.1.drv’’ done; 2 left
|   |   building of ‘/nix/store/knq494gr8l2ldq2l1n5hs227h8dsmq37-nss-cacert-3.31.drv’: init
|   |   building of ‘/nix/store/knq494gr8l2ldq2l1n5hs227h8dsmq37-nss-cacert-3.31.drv’: loading derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/knq494gr8l2ldq2l1n5hs227h8dsmq37-nss-cacert-3.31.drv’: have derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/knq494gr8l2ldq2l1n5hs227h8dsmq37-nss-cacert-3.31.drv’: done
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: waitee ‘building of ‘/nix/store/knq494gr8l2ldq2l1n5hs227h8dsmq37-nss-cacert-3.31.drv’’ done; 1 left
|   |   building of ‘/nix/store/bzvhbc586fayrp5savz2pgbm0iyp18x1-stdenv.drv’: init
|   |   building of ‘/nix/store/bzvhbc586fayrp5savz2pgbm0iyp18x1-stdenv.drv’: loading derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/bzvhbc586fayrp5savz2pgbm0iyp18x1-stdenv.drv’: have derivation
|   |   acquiring write lock on ‘/nix/var/nix/temproots/87548’
|   |   downgrading to read lock on ‘/nix/var/nix/temproots/87548’
|   |   building of ‘/nix/store/bzvhbc586fayrp5savz2pgbm0iyp18x1-stdenv.drv’: done
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: waitee ‘building of ‘/nix/store/bzvhbc586fayrp5savz2pgbm0iyp18x1-stdenv.drv’’ done; 0 left
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: woken up
|   |   building of ‘/nix/store/zjf6ppcfmlvrbqqq6zklsga12ll3scmz-bash-4.4-p12.drv’: goal destroyed
|   |   building of ‘/nix/store/ys1gdpj1y91cg8vf4ph7a3bx48dsphpg-git-2.14.1.drv’: goal destroyed
|   |   building of ‘/nix/store/bzvhbc586fayrp5savz2pgbm0iyp18x1-stdenv.drv’: goal destroyed
|   |   building of ‘/nix/store/knq494gr8l2ldq2l1n5hs227h8dsmq37-nss-cacert-3.31.drv’: goal destroyed
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: all inputs realised
|   |   building path ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’
|   |   added input paths ‘/nix/store/0fg0hsdc74c1b0s2m57kk0ib5mp3zb83-openssh-7.5p1’, ‘/nix/store/12mijs7c19ay8jnnd47wzkp410ha3r6n-gnutar-1.29’, ‘/nix/store/15mmqzifg0hdhxb2zbm99qk0sxz6rpx5-nss-cacert-3.31’, ‘/nix/store/1g4nzif4y5mz7j5k9b4whzz6z97kbd77-gcc-6.4.0-lib’, ‘/nix/store/2g7sy1wypsdz7765xz423549p7g52djh-openssl-1.0.2l’, ‘/nix/store/2hy5qbcl1yh5lrc25i6n5m1p8xd9rh15-linux-headers-4.4.10’, ‘/nix/store/3cn8dbaw0mwqhyddd7iawabhdwhbsyaa-paxctl-0.9’, ‘/nix/store/3i86a1bv4jzhh0s99787a637kq6ic6l9-diffutils-3.6’, ‘/nix/store/3rv8bjccvg4ml7x4bq59fqwhg554hyh1-findutils-4.6.0’, ‘/nix/store/47b2zmlwqvd9kdfz513ry8ma2id99916-libedit-20160903-3.1’, ‘/nix/store/54x6yrw5mppby8gk07bsw31v8aw448g1-acl-2.2.52’, ‘/nix/store/63dfz6sm3l3z0j2d9c7kk4ah7szyl644-ncurses-6.0-20170902’, ‘/nix/store/6684npam37a2mqwvlqfd17vvbv0lhgdx-curl-7.55.1’, ‘/nix/store/6ab1pnayxr4dr3k9b37w59mgwjcvwn6v-libssh2-1.8.0’, ‘/nix/store/6gxad5pia43zdcy0bc1ba8bxh7fhfpsg-zlib-1.2.11’, ‘/nix/store/6hq1smscdmm79ppzcxh6sapmgqfzaxdx-nghttp2-1.24.0-lib’, ‘/nix/store/81ikflgpwzgjk8b5vmvg9gaw9mbkc86k-compress-man-pages.sh’, ‘/nix/store/9ffr53k4dgkgw202lg5fsm66hyjdh6f9-gnused-4.4’, ‘/nix/store/a13mpdmxmg10245nbn84v893cy4pdr8p-patch-2.7.5’, ‘/nix/store/a7dxjj1kqc33mpmh7khfpbdxccvrghz3-perl-HTML-Tagset-3.20’, ‘/nix/store/a92kz10cwkpa91k5239inl3fd61zp5dh-move-lib64.sh’, ‘/nix/store/b1gkcsvp8a1mqm2dgbdqrf2l29s2p7z5-binutils-2.28.1’, ‘/nix/store/blyh706rm5svnfm215q149iagsb03vkp-bzip2-1.0.6.0.1-bin’, ‘/nix/store/bpvfrxapn3ip7fvr5jb5iwjp2mvjrjzg-gnugrep-3.1’, ‘/nix/store/bwax7l3946zn3l806727sd1fk8ssjgpv-glibc-2.25-49’, ‘/nix/store/d62ivqkhigy9bprax1330dhmxv55mjis-stdenv’, ‘/nix/store/dvx8ak0dwa9srwy36q1yvz4g85nfanl7-xz-5.2.3-bin’, ‘/nix/store/dxpjzyxbxr43w36pvcfmxs96d4g1j6zg-gawk-4.1.4’, ‘/nix/store/hsv895y5linmmyiq2mqhbgjp1ka9l1dr-xz-5.2.3’, ‘/nix/store/ibxds4qpdfx1f2hmrqi3bwpni9ijfr7i-perl-CGI-4.36’, ‘/nix/store/j1ly0zzpnzb8y8ry6k4vi4prk8vwq233-set-source-date-epoch-to-latest.sh’, ‘/nix/store/j1q2bm9qjrhr6q2z819dgzi7bmy3hhjq-audit-tmpdir.sh’, ‘/nix/store/jhgd9a2bxx3bb80cig4rwh20jqkmv1sj-binutils-2.28.1-lib’, ‘/nix/store/jl7gvzmn55vi1vj9ixp7vynzs282pimq-gnumake-4.2.1’, ‘/nix/store/jljnzwnwg9gj60k82ckbhr2v5q08jcxz-gcc-6.4.0’, ‘/nix/store/jr7l8xyyanakpqr954rvj58lyqal1vfb-builder.sh’, ‘/nix/store/k2h1nba3g9g4gknyr3npd69vkmgpbw7l-cracklib-2.9.6’, ‘/nix/store/l52acxgy4y3yn6wkr4c4fr5q9ny0i5ga-attr-2.4.47’, ‘/nix/store/m2yxakijvp9db943qf8a0qbchyyg3dsn-bash-4.4-p12’, ‘/nix/store/magv1gd9rrzlrmv7cgv2nsdpj7pqqd0f-coreutils-8.28’, ‘/nix/store/mfccjdkvrbs7pvwnqyvqhck1ajh5f7k8-nix-prefetch-git’, ‘/nix/store/mi9fgrz94516vyyp2fg7wan4q7qrv4k9-gcc-wrapper-6.4.0’, ‘/nix/store/mjjy30kxz775bhhi6j9phw81qh6dsbrf-move-docs.sh’, ‘/nix/store/nqx8hjlw731k1ycqv3za4hdi2fg0qbs7-pcre-8.40’, ‘/nix/store/nv83vcd5nvrkrjkr5y90zb2jg548d0dy-linux-pam-1.2.1’, ‘/nix/store/pqap7354dqnqw2ckpgicn2jmha5j8cq9-bzip2-1.0.6.0.1’, ‘/nix/store/q92c59h341d9xgkl2617pfwfhyl5lcs8-gzip-1.8’, ‘/nix/store/s5ibwbz0ad2s3kbfq5azc0dk11kp2w85-expand-response-params’, ‘/nix/store/s92rag4mn5i9vrv80pyiy672ibc7ijm5-perl-HTML-Parser-3.72’, ‘/nix/store/s96g9b8rl9ab13v1by7n661h0hzpbgsw-glibc-2.25-49-dev’, ‘/nix/store/scf9318hx6pad0wf0qx8a79h635aamfb-multiple-outputs.sh’, ‘/nix/store/sgmwb84x41c3q0i8h7h1n27jyd028j96-patchelf-0.9’, ‘/nix/store/srs62qc12vk0d1cm9zgnfc0dk6gg5ipp-git-2.14.1’, ‘/nix/store/v90p4bsjra1bjrgk8jgy0d29yasaad6h-glibc-2.25-49-bin’, ‘/nix/store/vgz4sms56q3dnwrvcpkjq26vs16gr646-strip.sh’, ‘/nix/store/w16d70yxr0miwpkb71qpq3lx70jh6z9f-perl-5.24.2’, ‘/nix/store/wd6qhr0arlz8z7hxsik9mcni92rsi2h5-expat-2.2.4’, ‘/nix/store/x8srildp8d5pcqfzzljcr55cwfdrhj6h-gettext-0.19.8’, ‘/nix/store/xfgjfjcq34fwfbfi1idp2nh81kpigch7-patch-shebangs.sh’, ‘/nix/store/z82dl6ialp166drqihzkz67nkl6w3l16-move-sbin.sh’, ‘/nix/store/zby9968vlkbm49l5rzsx677j1nkksg6z-ed-1.14.2’
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: woken up
|   |   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: trying to build
|   |   locking path ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’
|   |   lock acquired on ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839.lock’
|   |   building path(s) ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839’
|   lock released on ‘/nix/store/jfkvzdkd0vc83pzr5kbkh91r0cb0sxkr-devin-fcfc839.lock’
|   building of ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’: goal destroyed
error: a ‘x86_64-linux’ is required to build ‘/nix/store/glilggh7hx78jg2nx3y1j7rpvdzch6mf-devin-fcfc839.drv’, but I am a ‘x86_64-darwin’

[grahamc]

As an aside, with the closer integration between nix and the "build hook", it'd be nice if the debug messages were more helpful than just decline :) (ssh failed, wrong arch, busy, trusted users not setup right, missing feature X)

[mkg20001]

Added some debug logs in https://github.com/NixOS/nix/pull/3425

[nh2]

I would definitely appreciate more logs about this topic!

-vvv shows something on Nix 2.3.3:

considering building on remote machine 'ssh://static-haskell-nix-ci'
hook reply is 'decline'

But this still is not enough for me. Why doesn it not work?

I'm building with

--max-jobs 0 -A pkgs.haskellPackages.hspec --argstr system i686-linux -vvv

and have in my --builders file:

static-haskell-nix-ci         x86_64-linux - 4 - big-parallel

so perhaps it's that it doesn't want to build on x86_64-linux because I say i686-linux? No idea; it even fails if I replace the architecture in there.

I think the declineing should definitely say the reason why.

---

Edit for my memory: Workaround from the commandline:

nix-store -r --builders 'ssh://static-haskell-nix-ci x86_64-linux - - - big-parallel,kvm' --option builders-use-substitutes true --max-jobs 0

[domenkozar]

cc @arianvp since he got this error yesterday. Seems to be a problem with feature flags, architecture mismatch or lack of trusted-users permission.

[arianvp]

This indeed cost me at least 2 days of braintwisting. I was missing a big-parallel feature in my machine and this was happening when I tried building the kernel.

logging why something is declined would be very appreciated

[mkg20001]

I moved the debug logs now to this seperate PR: https://github.com/NixOS/nix/pull/3586

[bhipple]

This indeed cost me at least 2 days of braintwisting. I was missing a big-parallel feature in my machine and this was happening when I tried building the kernel.

Aha! This weekend I hit an issue where my build refused to compile LLVM on my enormous remote build machine, and since I couldn't figure out why I resorted to just waiting for it to compile locally on my laptop. If it's any consolation, your comment saved me a good bit of debugging :)

https://github.com/NixOS/nixpkgs/pull/86063

[bhipple]

It looks like this particular error has a nice proposal for improvement in @bburdette's https://github.com/bburdette/nix-error-proposal/blob/master/proposal.md#error-example

Looking forward to the day when this scenario produces this instead!

[blitz]

Today, I've also stumbled over this issue. There is really no good way to debug this. :(

I've already donated to https://opencollective.com/nix-errors-enhancement :)

[Ericson2314]

The C++ "none of the overloads of the function matches your call" errors I've been seeing a lot lately (:D :D :D) are a user interface. We might learn from them in e.g. sorting the builders by "distance" to the request, and perhaps only showing n closest matches.

[nh2]

PR #3897 helps with this.

[nh2]

I made a follow-up PR to add more details in #3927.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[anund]

I hit this attempting to remote build a system configuration. Helping with the search-ability a bit:

nix build -f '<nixpkgs/nixos>' config.system.build.toplevel --max-jobs 0 --builders 'ssh://nix-builder x86_64-linux - 8 -'

results in

error: unable to start any build; either increase '--max-jobs' or enable remote builds

The fix is to add big-parallel explicitly as mentioned above:

nix build -f '<nixpkgs/nixos>' config.system.build.toplevel --max-jobs 0 --builders 'ssh://nix-builder x86_64-linux - 8 - big-parallel'

[stale[bot]]

I marked this as stale due to inactivity. → More info

[athre0z]

Still an issue. Took me half an hour to figure out what was wrong here, with the relevant line buried in thousands of lines of logs. I should not have to crank up the verbosity so much to get that error report. The culprit turned out to be:

ignoring the client-specified setting 'builders', because it is a restricted setting and you are not a trusted user

IMHO this particular case should not trigger a log entry that only shows up on high verbosity levels, but a hard error: I was clearly using this tool wrong.