debootstrap not empty during stage 0

[knachte]

I set up a new wsl with ubuntu 24.04 today and started running into problems straight away:

During stage0 i get:

I: Extracting zlib1g...
W: Failure trying to run: chroot "/home/kristof/pi-gen.old/work/raspios/stage0/rootfs" /bin/true
W: See /home/kristof/pi-gen.old/work/raspios/stage0/rootfs/debootstrap/debootstrap.log for details
rmdir: failed to remove '/home/kristof/pi-gen.old/work/raspios/stage0/rootfs/debootstrap': Directory not empty
[12:22:20] bootstrap failed: please check /home/kristof/pi-gen.old/work/raspios/stage0/debootstrap.log

I checked the log mentioned above and at the end it contains:

chroot: failed to run command '/bin/true': Exec format error

I then went back to an older copy i had on another wsl running on 20.04, and that one suddenly started showing the same problem. I accidently nuked my 22.04 that i was using to build this morning, so can't go and check in that one anymore :(

I found an old issue here that mentioned the same problem from 2019, and checked that the patch in there is present in the current code: https://github.com/RPi-Distro/pi-gen/issues/248

Any idea what could be causing this?

[XECDesign]

The root of the issue is that your WSL environment isn't able to run the binaries of the architecture you're building for. That would normally be because binfmt isn't set up properly. On a normal Ubuntu install, as long as you have qemu-user-static and binfmt-support installed, it should just work.

What do you have in /usr/share/binfmts, /proc/sys/fs/binfmt_misc/ and what is the output of systemctl status binfmt-support?

[knachte]

I quickly installed the two packages you suggested and this is the output you requested.

kristof@ATLT-2202-1:~/pi-gen$ ls /usr/share/binfmts/
python3.12

kristof@ATLT-2202-1:~/pi-gen$ ls /proc/sys/fs/binfmt_misc/
WSLInterop  WSLInterop-late  python3.12  register  status

kristof@ATLT-2202-1:~/pi-gen$ sudo systemctl status binfmt-support
● binfmt-support.service - Enable support for additional executable binary formats
     Loaded: loaded (/usr/lib/systemd/system/binfmt-support.service; enabled; preset: enabled)
     Active: active (exited) since Thu 2024-05-02 14:18:56 CEST; 6min ago
       Docs: man:update-binfmts(8)
    Process: 752 ExecStart=/usr/sbin/update-binfmts --enable (code=exited, status=0/SUCCESS)
   Main PID: 752 (code=exited, status=0/SUCCESS)

May 02 14:18:56 ATLT-2202-1 systemd[1]: Starting binfmt-support.service - Enable support for additional executable binary formats...
May 02 14:18:56 ATLT-2202-1 systemd[1]: Finished binfmt-support.service - Enable support for additional executable binary formats.

Still with the same results when doing a build.

[XECDesign]

Okay, that's not what I'd expect if you have qemu-user-static installed, but I'm on 22.04. Let me check if anything relevant has changed in 24.04...

It may be /usr/lib/binfmt.d/ now. If you see files like qemu-aarch64.conf there, does restarting the binfmt-support service change what you see in /proc/sys/fs/binfmt_misc/?

[knachte]

/usr/lib/binfmt.d does indeed contain qemu-aarch64.conf and other architectures. Content of that file: :qemu-aarch64:M::\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/libexec/qemu-binfmt/aarch64-binfmt-P:OCPF

I restarted the service, no changes to the content of /proc/sys/fs/binfmt_misc/

[XECDesign]

If I search for binfmt wsl, there are many issues that come up, but I don't have a 24.04 WSL install handy to check which ones are actually applicable.

[knachte]

i'm seeing the same with my 20.04 install, but i'm not 100% certain that i ever built it on that one. I have a working one at home, i'll check that one when i get home later tonight if it's doing the same, but i'm not sure if it's a 20.4 or 22.4.

[XECDesign]

I use WSL a bit at home, so when I get the chance, I'll poke around a bit as well.

[knachte]

i've reinstalled a wsl 22.04 ubuntu and there it seems to work with no problems, so it seems that the problem is related to 24.04.

[XECDesign]

In my case, systemd doesn't work in WSL by default. Did you do something to enable it?

It looks like the systemd-binfmt service is meant to register everything in 24.04 instead of binfmt-support.

That service is meant to launch /usr/lib/systemd/systemd-binfmt. if I run that manuall, then it seems to work.

[XECDesign]

After updating wsl, systemd is working, and I've found this:

$ cat /usr/lib/systemd/system/systemd-binfmt.service.d/wsl.conf
# systemd breaks WSL interoperability by rewriting the binfmt interpreters configuration.

[Unit]
ConditionVirtualization=!wsl

So yeah, they intentionally disable it for whatever reason...

Edit: From what I gather, the reason is that they register other binfmt definitions (using wsl-binfmt.service) to make it possible to run .exe files from within wsl, and they don't want anything else overriding it (for example, wine or mono binfmt files, maybe), so they just break everything else instead 👍.

[knachte]

So basically, ubuntu 24.04 is a no go for building unless jumping trough a lot of hoops for the time being? I can stay on 22.04 without problem and this might be another good moment to bring up at work that they should let us dualboot linux.

[XECDesign]

The hoops shouldn't be too major. If you remember to run sudo /usr/lib/systemd/systemd-binfmt, it should work.

However, qemu has been known to be a bit flakey and cause issues in the past, so the best thing to do is to build natively on a pi with enough storage.

[knachte]

Ok, i'll give that a try at work tomorrow. Thanks for the help!

[XECDesign]

No worries. Good luck!

[knachte]

just for reference, at home i realised that on ubuntu 22.04 wsl i had not done a build yet on my main machine. I had to enable systemd in /etc/wsl.conf and restart the binfmt-support service which made /proc/sys/fs/binfmt_misc/ fill up, after which it would build. Before that i got the same result as on 24.04. At work the freshly installed 22.04 worked first time, so maybe older versions of 22.04 under wsl did not yet have systemd activated.

[knachte]

I just checked at work, the 22.04 and 24.04 that i installed yesterday both had systemd activated in /etc/wsl.conf. And even more.. the 24.04 that did not work yesterday did the build this morning without any problems, without any changes done to it. I'm guessing that it needed a complete restart of wsl before making everything active.

If you think this might be interesting enough to add to the readme.md i'll play with it a bit more this weekend and send a pr somewhere next week.

[XECDesign]

I think I'll add a check in build.sh before doing anything to verify that binfmt works as expected. If it doesn't, it should give some useful guidance rather than debootstrap's Exec format error ¯\_(ツ)_/¯

However, if it's not working out of the box, it generally indicates something wrong with the host system. I don't think it would be possible for us to document and track all of the potential issues when running in an unknown environment.

I think we'll say that native builds on a Raspberry Pi will work and are fully supported. But when users start introducing other distros, other kernels, VMs, containers, binfmt and emulators things start getting a bit fragile and those issues should go upstream to whichever component introduces it.

[XECDesign]

Should be "fixed" in https://github.com/RPi-Distro/pi-gen/commit/d87f764fcbc2e9bdbfd407d2cbf9b5faf5600df1