How to cross compile? - Githubissues

I'm trying to get cross-compilation working but without any luck so far. I tried every incantation of buildPlatform, hostPlatform, localSystem, crossSystem in either channels config or host's config, but I just can't get it to work. In particular:

{
  nixpkgs.buildPlatform = { config = "x86_64-unknown-linux-gnu"; };
  nixpkgs.hostPlatform = { config = "aarch64-unknown-linux-gnu"; };
}

This should cross-compile according to the nixpkgs documentation, but it doesn't.

Also this (unsure what the difference to above is supposed to be):

{
  nixpkgs.localSystem = { config = "x86_64-unknown-linux-gnu"; };
  nixpkgs.crossSystem = { config = "aarch64-unknown-linux-gnu"; };
}

Additionally, setting pkgs seems to have zero effect when using fup:

{
  nixpkgs.pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform
}

I also tried it in channels config:

{
    channels.cross = {
      input = nixpkgs; # input = nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform; doesn't work
      pkgs = nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform; # ignored?
      localSystem = { config = "x86_64-unknown-linux-gnu"; }; # ignored?
      crossSystem = { config = "aarch64-unknown-linux-gnu"; }; # ignored?
  };
}

Trying to read the code it looks like on is merely able to influence nixpkgs.config from the flake level, however all these options go into options directly under nipkgs.

Do you know of anyone successfully using fup for aarch64?

My aarch64 hosts either wrongly build as x86_64, or use qemu to build. The latter is extremely slow to a point it's unusable, and stopped working alltogether recently trying to build "grafana" (just stops doing anything without load). Any pointers?

Can you try something like this:

channels.cross.config = { 
      pkgs = nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform; // what does this do? botstrap nixpkgs?
      localSystem = { config = "x86_64-unknown-linux-gnu"; }; 
      crossSystem = { config = "aarch64-unknown-linux-gnu"; }; 
      };

This defines localSystem, crossSystem and pkgs on nixpkgs.config.*, but they are supposed to go onto nixpkgs.*. As far as I can tell, nixpkgs.pkgs should be a nixpkgs evaluation which gets any nixpkgs.overlays applied and then provided as pkgs to the nixos module system. When some nixos module refers to a pkgs, e.g. pkgs.grafana, they should get nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform.grafana. From the manpage:

If set, the pkgs argument to all NixOS modules is the value of this option, extended with nixpkgs.overlays, if that is also set. Either nixpkgs.crossSystem or nixpkgs.localSystem will be used in an assertion to check that the NixOS and Nixpkgs architectures match. Any other options in nixpkgs.*, notably config, will be ignored.

I think all the nixpkgs.* options in my host's modules have no effect because fup bootstraps nixpkgs beforehand and injects it into modules, but I got lost following the lib code.

@gytis-ivaskevicius I've done some experiments, and commenting out https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/lib/mkFlake.nix#L147 led to the system build picking up the relevant options (e.g. setting nixpkgs.pkgs to inputs.nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform led to it using cross compilation, as well as setting nixpkgs.localSystem = "x86_64-linux"; nixpkgs.crossSystem = "aarch64-multiplatform" appearently having the same effect. TBH I couldn't follow the whole expression, first of all I think there is no way options ? nixpkgs.pkgs is ever false, and https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/lib/mkFlake.nix#L161 only passes along nixpkgs.config, however localSystem and crossSystem go directly under nixpkgs, and are also to be passed directly on import (import nixpkgs { localSystem = "", crossSystem = "", config = {} }).

Also, updating https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/lib/mkFlake.nix#L212 to not blanketly inheriting system, but passing either crossSystem and localSystem or system also led to cross compilation correctly (nixpkgs assertion prevents passing system and localSystem at the same time).

All that said, I got some compilation issues, and cross compiling led to a full rebuild of all packages without getting any susbstitutions which defeats the purpose. I abandoned the project and I'm looking into ways to selectively cross-compile problem packages, and use qemu aarch64 emulation for the others (and benefit from substituions from nixos binary cache if I just want to use packages as is). Maybe some issues in fup still remain: e.g. I can't just use pkgsCross. because system==localSystem and consequently e.g. pkgsCross.aarch64-multiplatform.grafana in my nixos modules is in fact nixpkgs.legacyPackages.aarch64-multiplatform.pkgsCross.aarch64-multiplatform.grafana, so not cross-compiling at all. But I think there are avenues to work around this with constructing a special overlay that reaches back into inputs.nixpkgs.

I think I might have figured targeted cross compilation out:


{
…
      pkgsCrossAargh64 = import nixpkgs {
        localSystem = "x86_64-linux";
        crossSystem = "aarch64-linux";
        overlays = sharedOverlays;
      };
      channels.cross.input = nixpkgs;
      channels.cross.overlaysBuilder = _: [
        (final: prev: {
          inherit (self.pkgsCrossAargh64) grafana …;
        })
      ];

      hosts.archimedes.system = "aarch64-linux";
      hosts.archimedes.channelName = "cross";
…
}

thanks @pschyska for all the hints! I seem to have cross compilation working with the modifications your described here https://github.com/gytis-ivaskevicius/flake-utils-plus/issues/125#issuecomment-1320882037 .

All that said, I got some compilation issues, and cross compiling led to a full rebuild of all packages without getting any susbstitutions which defeats the purpose.

I think it is still useful and fup should not prevent building systems like that. I'll see if I can pick up your work and polish it for a PR.

I think I might have figured targeted cross compilation out:

that looks great! If I understand correctly nix will evaluate this and query the caches even on a non-aarach64 host without binfmt emulation, correct? It will probably fail later when assembling the system images unless everything (incl the system derivation, the systemd units, etc dir) depending on those packages is also explicitely listed for cross-compilation. But the approach of taking almost everything from the native caches and just cross-building the few needed packages and the resulting system looks very interesting!

Just to document this here:

I'll see if I can pick up your work and polish it for a PR.

I don't have the capacity to do anything about this currently.

I'm also not quite sure what the "right" way of implementing this would be. nixpkgs is switching away from specifying system as an argument when importing and encouraging setting nixpkgs.hostPlatform and nixpkgs.buildPlatform instead ^1. Maybe the channels themselves should stay "system-less"? Each host would need to specify the hostPlatform and buildPlatform then (or as a shortcut and backwards compatibility step keep host.system and set both to it as default). But I think that would break or require rework of the packages output of the flake as well.

that looks great! If I understand correctly nix will evaluate this and query the caches even on a non-aarach64 host without binfmt emulation, correct?

Sorry for the late reply; I'm actually not 100% clear what happens. I'm still using binfmt emulation, and I see some derivations that I pull in via this cross channel doing the main compilation with a cross compiler (e.g. aarch64-unknown-linux-gnu-gcc, which is a x86_64-linux binary producing aarch64 binaries), but using qemu-aarch64 for auxillary build steps (e.g. generating manpages with). I'm not sure it will work without binfmt-emulation in this case, or if I just have to pull in more packages via overlaysBuilder (besides grafana in my earlier example). IOW: Maybe the reason for this weirdness is that I construct franken-pkgs of some drvs coming from pkgsCross, where others come straight from pkgs 🙂 Maybe it would be worth to just use everything from pkgsCross and investigate the issues I got with that in more detail.

I'm mainly using this to save on build times. At the time, grafana didn't build for aarch64 in nixos-unstable, and it took over an hour to do it via qemu. I also have some custom packages or overlays with patches of nixpkgs packages that were very slow to build (neovim master, the bupstash backup tool, systemd-udevd, from the top of my head).

In general, the derivations are substitutable and are indeed substituted. Bear in mind that the cross drvs are different, e.g.:

nix-repl> inputs.nixpkgs.legacyPackages.aarch64-linux.grafana.outPath
"/nix/store/rqr2hac5gyw8x1l8a17cqzxi30dx7nxc-grafana-9.3.1"
nix-repl> inputs.nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform.grafana.outPath
"/nix/store/dgmcp0lpjsyz290vhd93vbhxnmc0l1qy-grafana-aarch64-unknown-linux-gnu-9.3.1"

I just built grafana with the config above, and it did build /nix/store/dgmcp0lpjsyz290vhd93vbhxnmc0l1qy-grafana-aarch64-unknown-linux-gnu-9.3.1, but not e.g. /nix/store/z7phzl9533c2pnb9x2n6v59cb43waf0j-glibc-aarch64-unknown-linux-gnu-2.35-224, that was substituted. The closure is:

/nix/store/122mrghlvpq5xp9addfyj0b3br6lnpac-tzdata-2022g                                        2101576
/nix/store/4xslbjwddi9h2xbxnrwnxdpwy75i9y64-iana-etc-20221107                                    569440
/nix/store/sxwpyh0zhg3dwf7izcgjnnya1lskvc2s-mailcap-2.1.53                                       112040
/nix/store/z7phzl9533c2pnb9x2n6v59cb43waf0j-glibc-aarch64-unknown-linux-gnu-2.35-224           41416392
/nix/store/dgmcp0lpjsyz290vhd93vbhxnmc0l1qy-grafana-aarch64-unknown-linux-gnu-9.3.1           321327352

Not sure why it built grafana-aarch64-unknown-linux-gnu-9.3.1 though, I don't have any changes to that packages. Maybe nix build doesn't substitute the drv but just its dependencies?

Not sure why it built grafana-aarch64-unknown-linux-gnu-9.3.1 though, I don't have any changes to that packages. Maybe nix build doesn't substitute the drv but just its dependencies?

I just tried nix profile install with different packages:

nix profile install --profile temp -L .#nixosConfigurations.archimedes.pkgs.fish:

Built fish-aarch64-unknown-linux-gnu, but substituted the dependencies (same result with nix profile install --profile temp -L nixpkgs#pkgsCross.aarch64-multiplatform.fish, ruling out other overlays in my flake). The closure is:

/nix/store/z7phzl9533c2pnb9x2n6v59cb43waf0j-glibc-aarch64-unknown-linux-gnu-2.35-224               41416392
/nix/store/9a8bzbrpcgy76wxw4q8x19nq2jxld4n9-aarch64-unknown-linux-gnu-stage-final-gcc-9.5.0-lib    48925640
/nix/store/6q1fkxz517bds0snj4x16dwvjm6g9q9z-db-aarch64-unknown-linux-gnu-5.3.28                    52889984
/nix/store/76bk7nvwzv4m6lp8s2n6vdjw5slk4m9b-bash-5.2-p15-aarch64-unknown-linux-gnu                 45002264
/nix/store/gb2xjzbmnh3kas3h5i4x8r893gw8l32w-libpipeline-aarch64-unknown-linux-gnu-1.5.6            41543656
/nix/store/kjn4cglgvlvdrza5qva0mgmafc9v348x-gzip-aarch64-unknown-linux-gnu-1.12                    41614488
/nix/store/gk2z1hivjdaq59sjpg8dd6gxfsrllmdw-pcre-aarch64-unknown-linux-gnu-8.45                    41964824
/nix/store/ijb936ylcfwh94zd905nsyjk5dvljzch-gnugrep-aarch64-unknown-linux-gnu-3.7                  42868784
/nix/store/j8siamvq2b8990l7pssjy5li2p2b635q-zstd-aarch64-unknown-linux-gnu-1.5.2                   42180840
/nix/store/qpq2zrksbk9q08l06lv4n3n8hi67inxh-zstd-aarch64-unknown-linux-gnu-1.5.2-bin               47440056
/nix/store/xi2x67mhw3mbrm48spdmdmr6qd6gq72r-groff-aarch64-unknown-linux-gnu-1.22.4                 61581496
/nix/store/0afr45arm65kwj8l5rfh5z577yq6dnjw-man-db-aarch64-unknown-linux-gnu-2.11.1                70566368
/nix/store/2iz92shwgh554p94rhfjc687r9cynias-tzdata-aarch64-unknown-linux-gnu-2022g                  2101576
/nix/store/4hagqmf8nca2jghci06n9y3j853y8y2k-attr-aarch64-unknown-linux-gnu-2.5.1                   41538704
/nix/store/2kcrca739c6k0ya4y2j83mgignbhs8z8-acl-aarch64-unknown-linux-gnu-2.3.1                    41682160
/nix/store/2v3fnashfzjh7zgzlgqf9hw4chnbn5yz-gettext-aarch64-unknown-linux-gnu-0.21                 62698984
/nix/store/42b5rkpyjzrf4ybfhzas3g0hhp0a58q6-bzip2-aarch64-unknown-linux-gnu-1.0.8                  41494168
/nix/store/4xg1xbq9yzzg1vs8srgzmpai5f8s3aqc-ncurses-aarch64-unknown-linux-gnu-6.3-p20220507        45470888
/nix/store/517msqz7zzw6fib5hcqgg83y1k4vp0wf-pcre2-aarch64-unknown-linux-gnu-10.42                  43196048
/nix/store/6431iv7q85y06j8gkx1w04dwij2bz9n8-gmp-with-cxx-aarch64-unknown-linux-gnu-6.2.1           49531296
/nix/store/70mmxia0xl0bvif9g486qq5xsjilpnjf-mailcap-aarch64-unknown-linux-gnu-2.1.53                 112040
/nix/store/7rgn80zjzrmnnm9k15x75jwpmz8crc2n-zlib-aarch64-unknown-linux-gnu-1.2.13                  41563568
/nix/store/93jriyl6pqgj2axbha6pciq9cd2yicav-libxcrypt-aarch64-unknown-linux-gnu-4.4.33             41648304
/nix/store/9fr6ygy7lmzya9n9z240m5vjz4f1m0wr-readline-aarch64-unknown-linux-gnu-8.2p1               46021760
/nix/store/aiw9796v9zzjxf2yxnw0ly3bzwvnvhl0-xz-aarch64-unknown-linux-gnu-5.4.0                     42223840
/nix/store/f1dc22l74lyk3fbi4p82rgjwkckpc9hl-gnused-aarch64-unknown-linux-gnu-4.9                   42229832
/nix/store/l7baa75yjx7fryyig8qdamkg23c6q4hk-coreutils-aarch64-unknown-linux-gnu-9.1                51378800
/nix/store/ir609az0hypdsm3kqkxv79hxrmgbcmzy-gdbm-aarch64-unknown-linux-gnu-1.23                    42374504
/nix/store/lbdijb3ivyh7ymndkkr2fc2lkaqdyfbs-expat-aarch64-unknown-linux-gnu-2.5.0                  41703032
/nix/store/p59j5i2ad48258nqsr2s2zr34s64skys-sqlite-aarch64-unknown-linux-gnu-3.40.1                42939384
/nix/store/yh5x8b77jhrrzmbdx6fj75gric5c8g4y-openssl-aarch64-unknown-linux-gnu-3.0.7                47613600
/nix/store/zja1dx4jpnmri5x5qknhjb72adlq0hav-libffi-aarch64-unknown-linux-gnu-3.4.4                 41494840
/nix/store/nxikay6ph9j0pni3ypwcdqyxklsa4zwr-python3-aarch64-unknown-linux-gnu-3.10.9              164617872
/nix/store/ydwzfbsbksl1s2d9df4p967drggyz4j7-gawk-aarch64-unknown-linux-gnu-5.2.1                   44836232
/nix/store/fj7k8wa273vvm1idm9skrwki2mh6waal-fish-aarch64-unknown-linux-gnu-3.6.0                  215960408

→ Still not clear to my why fish itself wasn't substituted.

nix profile install --profile temp3 -L nixpkgs#pkgsCross.aarch64-multiplatform.gimp

Substituted a lot (~3 GB 😲 ) but tried to rebuild around 160 packages, and ultimately failed with:

error: builder for '/nix/store/fsic6xf449lwccsn38sr1y4f5i7sw37x-libmypaint-aarch64-unknown-linux-gnu-1.6.1.drv' failed with exit code 1;
       last 58 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/6qsq7dddmmjlllks4inyn5p0bzb2ai3b-source
       > source root is source
       > patching sources
       > updateAutotoolsGnuConfigScriptsPhase
       > configuring
       >
       > I am testing that you have the tools required to build
       > libmypaint from git. This test is not foolproof.
       >
       > checking for libtool >= 1.5 ... Major version might be too new (2.4.7)
       > checking for autoconf >= 2.62 ... yes (version 2.71)
       > checking for automake >= 1.13 ... yes (version 1.16.5)
       > checking for intltool >= 0.40.1 ... yes (version 0.51.0)
       > checking for python ... yes (python)
       > WARNING: cannot find glib-2.0.m4 in aclocal's search path.
       >          You may see fatal macro warnings below.
       >          I looked in: /nix/store/m0x3gjgn720zp5plfr0bavm24iw0ip07-automake-1.16.5/share/aclocal /nix/store/z490g7746r1vfyp1rni3pz4nnqlcbynq-intltool-0.51.0/share/aclocal /nix/store/n162rpxjvh57qgmpdj90sg4b9f59k0sr-gettext-0.21/share/aclocal /nix/store/vkv46yln2g0vgvdn2nih1996d5g2p1f1-libtool-2.4.7/share/aclocal /nix/store/k79j16jangg0w4g5d18sz4ywqqwz6zah-aarch64-unknown-linux-gnu-pkg-config-wrapper-0.29.2/share/aclocal /nix/store/m0x3gjgn720zp5plfr0bavm24iw0ip07-automake-1.16.5/share/aclocal
       >          If these files are installed in /some/dir, set the
       >          ACLOCAL_FLAGS environment variable to "-I /some/dir",
       >          or append ":/some/dir" to ACLOCAL_PATH,
       >          or install /nix/store/m0x3gjgn720zp5plfr0bavm24iw0ip07-automake-1.16.5/share/aclocal/glib-2.0.m4.
       >
       > WARNING: cannot find glib-gettext.m4 in aclocal's search path.
       >          You may see fatal macro warnings below.
       >          I looked in: /nix/store/m0x3gjgn720zp5plfr0bavm24iw0ip07-automake-1.16.5/share/aclocal /nix/store/z490g7746r1vfyp1rni3pz4nnqlcbynq-intltool-0.51.0/share/aclocal /nix/store/n162rpxjvh57qgmpdj90sg4b9f59k0sr-gettext-0.21/share/aclocal /nix/store/vkv46yln2g0vgvdn2nih1996d5g2p1f1-libtool-2.4.7/share/aclocal /nix/store/k79j16jangg0w4g5d18sz4ywqqwz6zah-aarch64-unknown-linux-gnu-pkg-config-wrapper-0.29.2/share/aclocal /nix/store/m0x3gjgn720zp5plfr0bavm24iw0ip07-automake-1.16.5/share/aclocal
       >          If these files are installed in /some/dir, set the
       >          ACLOCAL_FLAGS environment variable to "-I /some/dir",
       >          or append ":/some/dir" to ACLOCAL_PATH,
       >          or install /nix/store/m0x3gjgn720zp5plfr0bavm24iw0ip07-automake-1.16.5/share/aclocal/glib-gettext.m4.
       >
       > configure.ac:238: warning: macro 'AM_GLIB_GNU_GETTEXT' not found in library
       > libtoolize: putting auxiliary files in '.'.
       > libtoolize: linking file './ltmain.sh'
       > libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4macros'.
       > libtoolize: linking file 'm4macros/libtool.m4'
       > libtoolize: linking file 'm4macros/ltoptions.m4'
       > libtoolize: linking file 'm4macros/ltsugar.m4'
       > libtoolize: linking file 'm4macros/ltversion.m4'
       > libtoolize: linking file 'm4macros/lt~obsolete.m4'
       > Writing mypaint-brush-settings-gen.h
       > Writing brushsettings-gen.h
       > configure.ac:96: installing './ar-lib'
       > configure.ac:94: installing './compile'
       > configure.ac:102: installing './config.guess'
       > configure.ac:102: installing './config.sub'
       > configure.ac:70: installing './install-sh'
       > configure.ac:70: installing './missing'
       > Makefile.am: installing './depcomp'
       > parallel-tests: installing './test-driver'
       > configure.ac:95: warning: The macro `AC_PROG_CC_C99' is obsolete.
       > configure.ac:95: You should run autoupdate.
       > ./lib/autoconf/c.m4:1659: AC_PROG_CC_C99 is expanded from...
       > configure.ac:95: the top level
       > configure.ac:252: error: possibly undefined macro: AM_GLIB_GNU_GETTEXT
       >       If this token and others are legitimate, please use m4_pattern_allow.
       >       See the Autoconf documentation.
       > /nix/store/r80vx3223fmdi0qc488x265l3bz79pcj-stdenv-linux/setup: line 129: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/fsic6xf449lwccsn38sr1y4f5i7sw37x-libmypaint-aarch64-unknown-linux-gnu-1.6.1.drv'.

I suspect the binary cache for pkgsCross is populated on a best-effort basis, and there will be many broken things in there because cross compilation is hard.

I'll probably keep building franken-pkgs for my raspberry and hope I don't hit these issues. Fingers crossed 😆

gytis-ivaskevicius / flake-utils-plus

How to cross compile? #125