Cannot cross-build images for a foreign architecture

[szlend]

It's currently not possible to cross-build images for a foreign architecture. Using nix2container on nixpkgs with a foreign crossSystem will attempt to build the nix2container-bin and jq binaries for the foreign system and then try to execute them at build time. You can work around this issue by using nix2container-bin based on pkgs.pkgsBuildHost. However this doesn't work correctly either, because arch is hardcoded to runtime.GOARCH.

You can work around both of these issues with a small wrapper:

{ go, jq, pkgsBuildHost, runCommand }:

# ...

buildImage = args:
  let
    arch = go.GOARCH;
    image = pkgsBuildHost.nix2container.buildImage args;
    nativeBuildInputs = [ jq ];
  in
  runCommand image.name { inherit arch nativeBuildInputs; } ''
    jq --arg ARCH "$arch" '.arch = $ARCH' ${image} > $out
  '';

Ideally this workaround wouldn't be necessary. We could fix this by:

1. Making nix2container-bin based on pkgs.pkgsBuildHost (this is equivalent to pkgs when crossSystem is not set).
2. Have buildImage pass a (new) --arch argument to nix2container-bin based on pkgs.go.GOARCH.

[nlewo]

Thank you for the report.

I agree with your proposal and contributions would be welcome ;)

[szlend]

After reviewing the source code, I can see that this is an issue all across the board and not just isolated to buildImage. I feel like we could restructure this a bit to make cross-compilation support easier to manage.

We could do the following:

1. Wrap nix2container with makeScopeWithSplicing'. This gives us a way to express different platform variants of our builders/derivations (e.g. buildHost, hostTarget, etc).
2. Wrap all builders/derivations into the callPackage pattern. This enables package splicing so nativeBuildInputs/buildInputs automatically pick the correct platform variants of our derivations when cross-compiling.
3. (Optional) Since the callPackage pattern can be pretty noisy, I would suggest extracting individual builders/derivations to their own nix files where it makes sense. Similar to how crane does it here: https://github.com/ipetkov/crane/tree/master/lib

For reference, I implemented proper splicing/cross-compilation support in crane recently: https://github.com/ipetkov/crane/pull/652

The alternative approach would be reviewing the source code and replacing certain instances of pkgs.<package> with pkgs.pkgsBuildHost.<package> and manually defining pkgsBuildHost variants of nix2container-bin/skopeo-nix2container. This is a more straightforward approach, but at the cost of a lot of cross-compilation specific noise.

@nlewo Which approach would you prefer? Both of these would be fully backwards compatible. I think the splicing approach would be nicer to work with, but it would be a larger change, cause big merge conflicts with open PRs and more difficult to review. Though the project has a pretty decent test coverage so we can be fairly certain we haven't broken something.

[szlend]

I opened a PoC PR in https://github.com/nlewo/nix2container/issues/139 to show what I mean.