Open LisannaAtHome opened 6 years ago
Doesn't this already work? I haven't tried it, but in principle there is nothing stopping a remote builder from forwarding builds itself.
There's at least one complication with this today: a proxy build server would have to have all of the supported features of anything it forwards to, even though that might not be the case. It must also be of the same system type.
An ideal user story for this would be something like a very lightweight proxy server which technically has a nix build environment and is setup to forward builds, but won't ever be used for any builds itself. I don't really see a way for a machine to forward a build it received without it also having some chance of being the thing that builds it. For build clusters with heterogeneous supported / mandatory features and systems, this won't work.
I marked this as stale due to inactivity. → More info
@edolstra I just tried to achieve sort-of the same thing, and I'm running into an issue.
Building for different architectures on the proxy server works just fine when executing nix
commands on there directly.
However, when I try to do a build from my local machine through that proxy, it fails with the following message.
Note, my local machine is of type aarch64-darwin
in this instance.
$ nix build -L --impure --expr '(with import <nixpkgs> { system = "aarch64-linux"; }; runCommand "foo" {} "uname -mo > $out")'
error: build of '/nix/store/nly18j4i88n7ppbfk9pkqy26j0qvvhxs-foo.drv' on 'ssh-ng://nixbld@proxyhost' failed: error: unable to start any build; remote machines may not have all required system features.
https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html
error: builder for '/nix/store/nly18j4i88n7ppbfk9pkqy26j0qvvhxs-foo.drv' failed with exit code 1
My local nix.conf
looks as follows.
allowed-users = root bryanhonof @staff
trusted-users = bryanhonof @staff
auto-optimise-store = false
builders = @/etc/nix/machines
builders-use-substitutes = true
download-attempts = 2
experimental-features = flakes ca-derivations impure-derivations nix-command recursive-nix repl-flake
log-lines = 25
max-jobs = auto
build-users-group = nixbld
Where /etc/nix/machines
stores the following.
ssh-ng://nixbld@proxyhost x86_64-linux,aarch64-linux,x86_64-darwin,aarch64-darwin /etc/ssh/ssh_host_ed25519_key 100 1 kvm,big-parallel,nixos-test,benchmark - c3…
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix. Do not edit it!
allowed-users = *
auto-optimise-store = false
connect-timeout = 10
cores = 0
extra-sandbox-paths =
max-free = 1000000000
max-jobs = 0
min-free = 128000000
require-sigs = true
sandbox = true
sandbox-fallback = false
stalled-download-timeout = 30
substituters = https://cache.nixos.org/
system-features = kvm big-parallel nixos-test benchmark
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
trusted-substituters =
trusted-users = @wheel
experimental-features = nix-command flakes
extra-trusted-users = nixbld
builders-use-substitutes = true
extra-platforms = x86_64-linux aarch64-linux x86_64-darwin aarch64-darwin
And it's /etc/nix/machines
file.
ssh-ng://nixbld@x86-intel-builder x86_64-linux /etc/ssh/ssh_host_ed25519_key 2 1 kvm,big-parallel,nixos-test,benchmark - c3…
ssh-ng://nixbld@aarch64-intel-builders aarch64-linux /etc/ssh/ssh_host_ed25519_key 2 1 kvm,big-parallel,nixos-test,benchmark - c3…
ssh-ng://nixbld@x86-darwin-builder x86_64-darwin /etc/ssh/ssh_host_ed25519_key 6 1 kvm,big-parallel,nixos-test,benchmark - c3…
ssh-ng://nixbld@aarch64-darwin-builder aarch64-darwin /etc/ssh/ssh_host_ed25519_key 6 1 kvm,big-parallel,nixos-test,benchmark - c3…
I already tried setting the cores
and max-jobs
config to a higher value than 0, which allows me to build for x86_64-linux
(the machine's architecture), but it doesn't proxy the build, it just builds it itself.
The nixbld
user has some ssh config to it, so it can only execute the nix-daemon
command. That's happening as follows.
{
services.openssh.extraConfig = let
command = pkgs.writeShellApplication {
name = "command";
runtimeInputs = [ pkgs.nix pkgs.coreutils ];
text = builtins.readFile ./nixbld-force-command.sh;
};
in ''
Match User nixbld
AllowAgentForwarding no
AllowTcpForwarding no
PermitTTY no
PermitTunnel no
X11Forwarding no
ForceCommand ${command}/bin/command
Match All
'';
}
The ForceCommand
is a simple bash script.
#!/usr/bin/env bash
if [[ "${SSH_ORIGINAL_COMMAND:=''}" != 'nix-daemon --stdio' ]]; then
echo "Hi ${USER}! You've successfully authenticated!";
echo "However, we do not provide shell access :( Sorry!";
exit 0;
fi
nix-daemon --stdio;
I believe I'm running into the following piece of code. https://github.com/NixOS/nix/blob/d00fe5f22559efc6f8b4b92eab537b08c0e43dee/src/libstore/build/worker.cc#L307-L325
I think the user story for many people sharing a cluster of remote nix builders is currently limited by the fact that remote building only works one level deep. i.e., if you have a cluster of build machines that you want to share with many Nix users, each user has to maintain their own copy of the list of each machine, and each user has to have an account on each of the machines (or otherwise share an account and SSH key).
This could be improved with allowing remote building to forward builds to machines that are themselves configured as remote builders on the currently selected remote builder. This would allow you to create clusters of build machines where in machine A has B, C, and D configured as remote builders, and anyone who wants to use the large cluster for remote building only has to know about and have access to machine A.