Always cross compile

[Ericson2314]

Exherbo, another linux distribution that has done good work with cross-compiling, always builds cross compilers---i.e. --host --build and --target are always passed to gcc's build system even if some of those platforms are the same: http://git.exherbo.org/arbor.git/tree/packages/sys-devel/gcc/gcc.exlib#n187. The big advantage here is by using the cross-compiling code path in all cases, there's less to maintain, and actual cross compilation is less likely to rot assuming native compilation will always be better tested.

In https://github.com/NixOS/nixpkgs/pull/21268 (specifically https://github.com/NixOS/nixpkgs/pull/21268/commits/633feb4e39a24c7da9ff4ae1b5f4219e0c2a1bb6 but I'll probably rebase at some point breaking that link), I introduce always-defined buildPlatform hostPlatform and targetPlatform for the same reasons (I keep around nullable crossSystem and not-always present stdenv.cross for compatibility). So if/when that PR is merged would be a good time to tackle this. that PR will probably be closed as a bunch else happened separately, including always defining buildPlatform hostPlatform and targetPlatform and removing crossSystem and stdenv.cross altogether.

Naturally this would be a gcc-caused mass rebuild on Linux. Darwin shouldn't be as affected as LLVM, by default, includes all the targets we care about.

[Ericson2314]

@DavidEGrayson's work in https://github.com/DavidEGrayson/nixcrpkgs/ should greatly accelerate this!

[Ericson2314]

@cleverca22 points out https://github.com/taktoa/arcane-chat/blob/master/default.nix has a bunch of hacks needed to get windows to work. Would be good to keep these things in mind.

[Ericson2314]

OK, hope this actually happens this cycle :crossed_fingers:.

[wmertens]

This will be seriously amazing!

So after a quick look, this change requires changing all occurrences of crossSystem to hostPlatform or targetPlatform? Is there no way to guess the correct value at evaluation time?

In other words, it's a huge single-shot change and after that it will just work?

If so, I propose we declare a week "cross-compile week" and only merge changes that make this work. Once everything builds on staging we merge to master and we're done?

Or is there more to it? Will there be things that break?

[Ericson2314]

The crossSystem vs hostPlatform things are actually all taken care of, now! What's left is:

• [ ] #36867 LLVM-based cross compilation
• [ ] Change configureFlags on binutils and gcc to be [ "host" "build" "target" ]. Watch out for this arm (armv5 I think it is?) case where then things fail to build. Also do for ghc and any other compiler where we have configureFlags set today. Also make the various cases of targetPrefix unconditionally hostPlatform.config "-". (Must be done with the previous.

[Ericson2314]

stdenv.cross and `crossSystem are already sufficiently purged!

The first thing to do is just make all targetPrefixes unconditional, and also --target for compilers that might use it. Then start passing --build and --host to everything else. @lheckemann found a safer alternative for that second step.

[lheckemann]

Said alternative is setting the build_alias, host_alias and target_alias environment variables. However, I'm not sure if this is really a desirable solution, both because it'll only work for autotools-based configure scripts and because I'm not sure this way of setting the platforms is actually considered a public/documented/stable interface by autotools.

[wmertens]

So when everything is cross-compiled, will the hashes of something cross-compiled on platform A to platform T be the same as platform B to T?

On Tue, Mar 27, 2018 at 1:15 PM Linus Heckemann notifications@github.com wrote:

Said alternative is setting the build_alias, host_alias and target_alias environment variables. However, I'm not sure if this is really a desirable solution, both because it'll only work for autotools-based configure scripts and because I'm not sure this way of setting the platforms is actually considered a public/documented/stable interface by autotools.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/21471#issuecomment-376487828, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWlkAUjoLYAcY44QQdUpda1jG27RR7ks5tih9pgaJpZM4LXFz4 .

[lheckemann]

No. The derivation hashes will be different, and as a result the output hashes will also be different. In an ideal scenario the only differences between cross-compiled and natively-compiled outputs will be the store paths, but I highly doubt that we'll achieve that in practice.

[wmertens]

Well, maybe we can achieve that with a CAS store, but only if the bytecode is 100% the same… It would be great if we could exclude the host platform from the input hashes…

On Tue, Mar 27, 2018 at 2:04 PM Linus Heckemann notifications@github.com wrote:

No. The derivation hashes will be different, and as a result the output hashes will also be different. In an ideal scenario the only differences between cross-compiled and natively-compiled outputs will be the store paths, but I highly doubt that we'll achieve that in practice.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/21471#issuecomment-376500123, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWloXVPFtZbrT_4rK4ySTA-nSuIEG_ks5tiirYgaJpZM4LXFz4 .

[Ericson2314]

@wmertens Yes that is a goal, but it will be very hard. CAS / intentional store is necessary, but not sufficient, as the input binaries for the build tools would not be anywhere near bit-for-bit the same.

[Also, you mean build platform not host platform, BTW. The names are indeed awful.]

[lheckemann]

@dezgeg raised the concern of tests not being run when cross-compiling. Does exherbo maybe have a solution that we can steal?

[eternaleye]

@lheckemann: My experience on Exherbo has been that tests run just fine when build == host (build == run), and would likely run just fine any time host (run) is runnable (such as i686 or x32 on an x86_64 builder). That could be further expanded using binfmt_misc and qemu-user.

[Ericson2314]

I can't find it well on my phone, but the answer must be in https://git.exherbo.org/paludis/paludis.git/

[eternaleye]

This is the relevant bit, but is overconservative - it just checks for build == host (build == run); though it misnames 'host'/'run' as 'target'.

[Ericson2314]

@Eternaleye err we mean how are (autoconf-based) configure scripts invoked. I see default_src_configure but not where the various env vars it uses are defined.

[eternaleye]

@Ericson2314: All the DEFAULT_SRC_CONFIGURE_* variables are set by individual packages. They aren't the _defaults for src_configure_; they allow packages to parameterize the _default implementation of src_configure_.

You probably want to look at the definition of econf instead.

[Ericson2314]

@eternaleye https://git.exherbo.org/paludis/paludis.git/tree/paludis/repositories/e/ebuild/exheres-0/build_functions.bash?h=cross#n155 huh so --build and --host indeed are always passed if CBUILD and CHOST are defined. Do you all indeed never have configure run programs then? Brave!

[eternaleye]

That's a work in progress. The most problematic cases are actually things like gobject-introspection, which is inherently incompatible with cross-compilation at the architectural level. Things like that work in practice when build == host (build == run), but fail when "really" cross-compiling. The ability to ban commands in the environment is the main tool used to fix such issues where possible - for example, we ban unprefixed pkg-config so we can learn what needs patched to respect --host and call ${HOST}-pkg-config instead.

Edit: Also, remember that configure is just shell templated by m4. Exherbo can't "never have configure run programs" - that's in upstream's hands. We can, however, try to detect when they do something incorrect so we can send them a patch - to prefix pkg-config, to use pkg-config instead of archaic foo-config programs, to use the correct include dir, or other such issues.

[dezgeg]

There was a misunderstanding, by "tests" I mean this particular autoconf macro which can spew the "cannot run test program while cross compiling" error:

# AC_RUN_IFELSE(PROGRAM,
#               [ACTION-IF-TRUE], [ACTION-IF-FALSE],
#               [ACTION-IF-CROSS-COMPILING = RUNTIME-ERROR])
# ----------------------------------------------------------
# Compile, link, and run. Requires that the compiler for the current
# language was checked for, hence do not use this macro in macros looking
# for a compiler.
AC_DEFUN([AC_RUN_IFELSE],
[AC_LANG_COMPILER_REQUIRE()dnl
m4_ifval([$4], [],
     [AC_DIAGNOSE([cross],
             [$0 called without default to allow cross compiling])])dnl
AS_IF([test "$cross_compiling" = yes],
  [m4_default([$4],
       [AC_MSG_FAILURE([cannot run test program while cross compiling])])],
  [_AC_RUN_IFELSE($@)])
])

What will happen on those calls? And in general, how does the cross_compiling autoconf variable get set in Exherbo? Is it just the default autoconf behaviour of setting it to yes if both --build and --host were specified and they were not equal, or no otherwise?

[eternaleye]

@dezgeg: AFAIK Exherbo does nothing special there, but I'd need to check. Speaking from experience, AC_RUN_IFELSE isn't especially common for packages to use explicitly, and can often be patched out (or have fallbacks patched in) easily enough - most upstreams take cross fixes well. We do add fallbacks where other autoconf macros use it though.

[Ericson2314]

Neat patch!

[dezgeg]

Indeed that patch looks like it solves many of those problems in practice. But my general point stands, aka even if you do "cross compilation" with e.g. both build and host set to x86, you still wouldn't detect those problems, you still would have to do e.g. and x86 -> ARM cross compile to actually detect them, right?

[eternaleye]

@dezgeg Yes; there's really no feasible way to do static analysis of autotools-based buildsystems.

[Gaelan]

I'm running into this when trying to compile for aarch32 on aarch64. I've been sending PRs to set configurePlatforms = ["host" "build"]; on a per-package basis, but is there anything stopping us from doing it globally at this point?

[wmertens]

Let's put it in staging and watch the world burn :)

[stale[bot]]

I marked this as stale due to inactivity. → More info

[Ericson2314]

Still pertinent, and in fact https://github.com/NixOS/nixpkgs/pull/87909 is getting new attention.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[sternenseemann]

We did make some progress towards this: #123419, #119625, … (?)

[wmertens]

One future concern: would we be ok with declaring cross-compiled binaries as equivalent, no matter what the build platform was?

If so, provided we have a trusted mapping (being looked at with the intensional efforts), how would we go about generating all the input hashes that map to the same build?

[Ericson2314]

@wmertens I don't think we should calculate them up front. The space of possible build platforms is very, very large.

[wmertens]

@Ericson2314 another option would be to somehow replace the build platform input hash with a placeholder when calculating the input hash? Then all cross-compiling build tools for a target platform would map to the same dependency input hash?

[stale[bot]]

I marked this as stale due to inactivity. → More info

[Lahvuun]

I'll use this chance to mention that I've been looking into cross compilation with Nix, and my conclusion is that the build platform shouldn't go into an input hash at all. It makes no sense to have two bit-for-bit identical packages with different hashes simply because one was built natively and another one cross compiled. The host platform (on which the package will be used) should be the one included in the input hash. And for packages that have no native code (for example text files, like configuration or Python programs) platform shouldn't matter at all.

If you get rid of that, then the other source of difference is the compiler. Here, you basically want native and cross compilers built from the same source with the same codegen options to have different store paths, but affect input hashes in the same way. I don't think this is possible without extending Nix, and even then this kind of rule bending feels like it goes against the core Nix philosophy.

[Ericson2314]

@lahvuun I suspect with CA derivations we could do "cache cheating as a plugin" to allow people to experiment with mixing builds without baking in logic in Nix contra to our morals.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixlang-how-do-you-find-all-uses-of-a-declaration/18369/13

[Mindavi]

This PR is just about done now: https://github.com/NixOS/nixpkgs/pull/87909

I'm thinking it might be merged just after branch-off of 22.05?

[sternenseemann]

Only a step towards it, however, since prefixing the compiler by default is still missing.

[Mindavi]

Yeah, that'll be an interesting exercise