Processes started by cross emulator can't exec other binaries (?)

[sternenseemann]

Describe the bug

Consider the following expression where koka is a shell script created using wrapProgram:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

let
  kokaHelloWorld = writeText "hello.kk" ''
    fun main() {
      println("Hello, World!");
    }
  '';

  crossTest = crossPkgs:
    crossPkgs.runCommand "koka-cross-test" {} ''
      ${crossPkgs.stdenv.hostPlatform.emulator crossPkgs.buildPackages} \
        ${crossPkgs.koka}/bin/koka ${kokaHelloWorld} \
        > $out
    '';
in

crossTest pkgsCross.aarch64-multiplatform

This will fail with the following error message:

Error while loading /nix/store/dnv68agszwxm9rw4njsjl2di3fvvvzph-koka-2.1.1-aarch64-unknown-linux-gnu/bin/koka: Exec format error

At first I suspected that qemu can't deal with shebang scripts for some reason since swapping out koka for .koka-wrapped works (at least the binary runs, but fails due to missing certain environment variables). But the issue seems to be even worse than that:

• Using ${crossPkgs.bash} -c ${crossPkgs.koka}/bin/koka doesn't work
• Using ${crossPkgs.bash} -c ${crossPkgs.koka}/bin/.koka-wrapped doesn't work

Seems like it's pretty much impossible for a process started by the emulator to call exec without running into executable format issues. Not sure if this is fixable without launching a full VM?!

Note that this makes most sense to test against #119468.

Notify maintainers

cc @Ericson2314 @matthewbauer @siraben

[Ericson2314]

Well, can't say I'm too surprised. I googled "binfmt_misc" and "child"/"fork" and didn't see anything obvious either.

[domenkozar]

According to Qemu docs we're using qemu-${arch} which is user emulation, but you could try with qemu-system-${arch} which is full emulation but slower.

I'm new to all this so I might be wrong :)

[r-burns]

Basically the execve call goes straight to the kernel, which doesn't know it's supposed to be emulating.

Unfortunately there is no upstream support for this, but there are downstream patches which fix this: https://github.com/pld-linux/qemu/blob/9793852df1b6132791fd7995b4eedd04d6c7948e/qemu-user-execve.patch

[Mindavi]

I don't think we can really solve this in nixpkgs. However, there's a new tool called makeBinaryWrapper in our toolbox which may help with this situation.