MacOs: `nix build` ignores using a remote builder for a different system unless `--max-jobs 0` is passed

[Tristano8]

Describe the bug

nix build doesn't use a remote builder, even when passed explicitly via --builder, unless explicitly telling it to ignore local builds. It seems like the nix scheduler thinks that it can build an x86_64-linux derivation locally, then at build time discovers that it cannot actually do that

Steps To Reproduce

1. On a silicon Mac, set up a remote builder in /etc/nix/nix.conf for x86_64-linux

eg:

builders = ssh://mybuilder x86_64-linux - 6
builders-use-substitutes = true

3. Run nix build -Lv --rebuild nixpkgs#legacyPackages.x86_64-linux.hello
4. See the following error:

   error: a 'x86_64-linux' with features {} is required to build '/nix/store/amplsdcav0g3qp9srkkfnqlkr8zc3sn8-hello-2.12.1.drv', but I am a 'x86_64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

5. Repeat step 3 with --max-jobs 0
6. Observe no error

Expected behavior

The result from step 6 should occur without passing --max-jobs

nix-env --version output nix-env (Nix) 2.18.1

Additional context

This occurs on multiple silicon machines (M1 and M3) and across multiple nix versions (2.21.1 and 2.18.1 at least).

Full error logs:

error-log.txt

My nix show-config:

accept-flake-config = false
access-tokens = <REDACTED>
allow-dirty = true
allow-import-from-derivation = true
allow-symlinked-store = false
allow-unsafe-native-code-during-evaluation = false
allowed-impure-host-deps = /System/Library /bin/sh /dev /usr/lib
allowed-uris = 
allowed-users = *
auto-allocate-uids = false
auto-optimise-store = false
bash-prompt = 
bash-prompt-prefix = 
bash-prompt-suffix = 
build-hook = /nix/store/6zi5fqzn9n17wrk8r41rhdw4j7jqqsi3-nix-2.18.1/bin/nix __build-remote
build-poll-interval = 5
build-users-group = nixbld
builders = ssh://remote-builder x86_64-linux - 6
builders-use-substitutes = true
commit-lockfile-summary = 
compress-build-log = true
connect-timeout = 0
cores = 12
darwin-log-sandbox-violations = false
diff-hook = 
download-attempts = 5
download-speed = 0
eval-cache = true
experimental-features = flakes nix-command
extra-platforms = aarch64-darwin
fallback = false
flake-registry = https://channels.nixos.org/flake-registry.json
fsync-metadata = true
gc-reserved-space = 8388608
hashed-mirrors = 
http-connections = 25
http2 = true
id-count = 128
ignore-try = false
impersonate-linux-26 = false
keep-build-log = true
keep-derivations = true
keep-env-derivations = false
keep-failed = false
keep-going = false
keep-outputs = false
log-lines = 10
max-build-log-size = 0
max-free = 2147483648
max-jobs = 1
max-silent-time = 0
max-substitution-jobs = 16
min-free = 1073741824
min-free-check-interval = 5
nar-buffer-size = 33554432
narinfo-cache-negative-ttl = 3600
narinfo-cache-positive-ttl = 2592000
netrc-file = /etc/nix/netrc
nix-path = /Users/hughdavidson/.nix-defexpr/channels nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixpkgs /nix/var/nix/profiles/per-user/root/channels
plugin-files = 
post-build-hook = 
pre-build-hook = 
preallocate-contents = false
print-missing = true
pure-eval = true
require-drop-supplementary-groups = false
require-sigs = true
restrict-eval = false
run-diff-hook = false
sandbox = false
sandbox-fallback = true
sandbox-paths = /System/Library/Frameworks /System/Library/PrivateFrameworks /bin/bash /bin/sh /private/tmp /private/var/tmp /usr/lib
secret-key-files = 
show-trace = false
ssl-cert-file = /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt
stalled-download-timeout = 300
start-id = 56930
store = auto
substitute = true
substituters = <REDACTED>
sync-before-registering = false
system = x86_64-darwin
system-features = benchmark big-parallel nixos-test
tarball-ttl = 3600
timeout = 0
trace-function-calls = false
trace-verbose = false
trusted-public-keys = <REDACTED>
trusted-substituters = 
trusted-users = hughdavidson
use-case-hack = true
use-registries = true
use-sqlite-wal = true
use-xdg-base-directories = false
user-agent-suffix = 
warn-dirty = true

[arianvp]

This also affects nix-build.

 $ nix-build -A systemd --builders 'ssh://utm.local aarch64-linux' --system 'aarch64-linux'

 error: builder for '/nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv' failed with exit code 126;
       last 1 log lines:
       > /nix/store/1fzg4cl3k2n9yq80vg6y1vcmvx3qm682-bash-5.2p26/bin/bash: /nix/store/1fzg4cl3k2n9yq80vg6y1vcmvx3qm682-bash-5.2p26/bin/bash: cannot execute binary file
       For full logs, run 'nix log /nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv'.
error: 1 dependencies of derivation '/nix/store/1h92zc3pa1v566r2s0za9dfbq9krgwra-python3-3.11.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/i01isqxbhsr286da2k6f48bd6i1xssg0-python3-3.11.9-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vnlxxv5fh2hc78hl0whvr25ay5pwwwdd-systemd-255.6.drv' failed to build
arian@Arians-MacBook-Pro nixpkgs % cat /nix/store/n2qn0hq1mdyyfplphymv9l7kq9x2pd25-builder.pl.drv

doesn't work

whilst

 nix-build -A systemd --builders 'ssh://utm.local aarch64-linux' --system 'aarch64-linux' --max-jobs 0

does