Trouble getting the remote builder to work on macOS

[dhess]

I've got the ssh remote builder container running in Docker on my macOS box. I can ssh into the container from the macOS host, and I've created and uploaded a signing key for remote builds.

However, when trying to use the container from macOS as a remote builder for Linux targets, I'm running into nixpkgs assertions at various places. For example, when I attempt to build one of my Haskell projects for Linux from macOS, using the remote builder container, after sourcing your remote-build-env script, I get the following:

% nix-build '<nixpkgs>' default.nix --argstr system x86_64-linux
error: assertion failed at /Users/dhess/git/nixpkgs/pkgs/top-level/all-packages.nix:6087:19

I'm using nixpkgs from git; the line shown in the error message above refers to the cross-compiling check:

  binutilsCross = assert crossSystem != null; lowPrio (forceNativeDrv (
    if crossSystem.libc == "libSystem" then darwin.cctools_cross
    else binutils.override {
      noSysDirs = true;
      cross = crossSystem;
    }));

I'm having a similar problem trying to use the ssh container with nixops to provision a NixOS physical host from macOS. nixops breaks as soon as it tries to build a kernel for the target host, though at least in this case it's finding my local remote-systems.conf file:

% nixops deploy -d builder-vm
building all machine configurations...
using predefined remote systems file: /Users/dhess/.nixconf/remote-systems.conf
error: assertion failed at /Users/dhess/git/dhess-nixos/nixpkgs/pkgs/os-specific/linux/kernel/generic.nix:32:1
(use ‘--show-trace’ to show detailed location information)
error: unable to build all machine configurations

Is a local nix-daemon a pre-requisite for using a remote builder? I'm not running one at the moment as I'm not using your nix-darwin repo and I've never needed it for my macOS work.

My remote-systems.conf file looks like this:

nix-docker x86_64-linux ~/.ssh/dhess@home.dhess.net_id_ed25519_20151127.pub 2

[LnL7]

You don't need the nix-daemon for remote builds, does a regular build work?

$ nix-build '<nixpkgs>' -A hello --argstr system x86_64-linux
$ nix-build '<nixpkgs>' -A hello --argstr system x86_64-linux --check

[dhess]

The former does (but it just downloads the nar file from the binary cache); the latter doesn't:

% nix-build '<nixpkgs>' -A hello --argstr system x86_64-linux --check
fetching path ‘/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz’...

*** Downloading ‘https://cache.nixos.org/nar/0yc68z03plkyms2vgr4f8byya75zcgqzbqsmbvazv8cf1fi70vdz.nar.xz’ to ‘/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  709k  100  709k    0     0   488k      0  0:00:01  0:00:01 --:--:--  489k

...

*** Downloading ‘https://cache.nixos.org/nar/0rd1p8zicpr97wnb11l0nab19130prdjnhz2xdcc19igkf16hi53.nar.xz’ to ‘/nix/store/aqafkcxpw23kk9fp2bavyczck4nnh9m1-stdenv’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8124  100  8124    0     0  11261      0 --:--:-- --:--:-- --:--:-- 15836

checking path(s) ‘/nix/store/7bqd18jfn63pqkcs78lcas5mcp10vmpy-hello-2.10’
error: a ‘x86_64-linux’ is required to build ‘/nix/store/p1kilmvgc7bjk2b03958mbj17qbcirqp-hello-2.10.drv’, but I am a ‘x86_64-darwin’

Here are the relevant environment variables:

NIX_BUILD_HOOK=/Users/dhess/.nix-profile/libexec/nix/build-remote.pl
NIX_CONF_DIR=/Users/dhess/.nixconf
NIX_REMOTE_SYSTEMS=/Users/dhess/.nixconf/remote-systems.conf
NIX_PATH=nixpkgs=/Users/dhess/git/nixpkgs
NIX_CURRENT_LOAD=/tmp/nix/current-load

My slightly modified version of remote-build-env looks like this:

#!/usr/bin/env bash

mkdir -p /tmp/nix/current-load
chmod a+rwX /tmp/nix/current-load

export NIX_BUILD_HOOK="${HOME}/.nix-profile/libexec/nix/build-remote.pl"
export NIX_REMOTE_SYSTEMS="${NIX_CONF_DIR}/remote-systems.conf"
export NIX_CURRENT_LOAD="/tmp/nix/current-load"

My nixpkgs repo is up-to-date with https://github.com/NixOS/nixpkgs/commit/13d6681ce72ac265f03013247f60ab3d001779e0 and my nix-env version is 1.11.6.

Thanks for the assistance!

[dhess]

Here is the verbose output from the failing command: https://gist.github.com/dhess/465e84c60e517ca98252870bfdd62a29

[LnL7]

Where did you put the signing keys? If you set NIX_CONF_DIR they might also need to be in there.

[dhess]

The signing keys are also in NIX_CONF_DIR.

[dhess]

@LnL7 are you able to get the ssh remote builder to work as expected on your Mac, running your nix-darwin environment?

[LnL7]

Figuring out why it doesn't work is quite tricky, there are a number of things that fail silently. I did just notice that you seem to be using your public key in the remote-systems.conf, I think that might be the problem here.

[dhess]

Hmm. I did replace the key on the ssh container with my own per the instructions in this repo's README.md, and I can ssh into the container as root using that key.

[LnL7]

Yes, but the build hook uses the private keys specified in NIX_REMOTE_SYSTEMS while you configured it with the .pub file. I expect that causes the builder to be ignored silently.

[dhess]

Good catch! Unfortunately, after fixing remote-systems.conf to use my private key, I'm still getting the same failures.

[dhess]

I think it's pretty likely that this is some weird issue with nix-build and/or my environment, and not the ssh Docker image. You can close this issue if you like. Otherwise, I'll report back here when I figure out what's going on. I would expect that at most a change is needed to the README.

[LnL7]

Perhaps https://github.com/LnL7/nix-docker/pull/5 helps?