systemd-nspawn issue on debian 12 guest / M2 MacBook Pro host

[ljames8]

ALSO, if systemd-nspawn fails for you please open an issue on this GitHub. I'd like to do some testing, but I'm having trouble finding one that fails!

Hi, here is the issue I encountered when running sdm --explore <raspios-image-file>.img from my debian 12 vm (distro is from https://mac.getutm.app/gallery/debian-12) running on my M2 MacBook pro with UTM

debian@debian:~$ sudo sdm --explore my-image.img
* Enter IMG 'my-image.img'
execv(/bin/bash) failed: Exec format error
umount: /mnt/sdm/boot unmounted
umount: /mnt/sdm unmounted

As sdm --chroot --explore <raspios-image-file>.img also failed with a "/bin/bash: /bin/bash: cannot execute binary file" error, I noticed the actual issue was the missing /proc/sys/fs/binfmt_misc/qemu-arm which seemed not be added despite running update-binfmts

I managed to workaround thanks to the binfmt_misc tip found in this discussion about arm32 qemu support from apple silicon: https://github.com/containers/podman/discussions/19329#discussioncomment-6528572

It looks like after running this (after each vm reboot)

root@debian:/home/debian# echo ":qemu-arm:M:0:\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:F" > /proc/sys/fs/binfmt_misc/register

I am now able to successfully explore my image

debian@debian:~$ sudo sdm --explore my-image.img
* Enter IMG 'my-image.img'
root@sdm:/#

Note: however sdm --chroot --explore <raspios-image-file>.img still fails with the same error as above, which I don't understand.

[gitbls]

Interesting problem! Just to confirm, your debian 12 VM is 64-bit and the IMG you're working with is a 32-bit RasPiOS IMG?

I don't have a Mac of any type for testing. Are you up for some trial and error debugging?

Thx!

[ljames8]

That’s right: 64bit vm and 32bit IMG

I sure can, let me know what I can do

[gitbls]

First thing I'd like to work out is how to identify this scenario. So, a couple of questions to get started:

• Output from Debian VM uname -a
• Output from Debian VM file /bin/ls
• Are you aware of any way to tell from inside the Debian VM that it's running on a Mac? I did a quick look at UTM docs but saw nothing obvious.

Once I get your input on the above, I think I can sort out how to automagically make it work. We'll see 🤔

[ljames8]

Here you go:

debian@debian:~$ uname -a
Linux debian 6.1.0-18-arm64 #1 SMP Debian 6.1.76-1 (2024-02-01) aarch64 GNU/Linux
debian@debian:~$ file /bin/ls
/bin/ls: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=9f127c37a4c459cf01639f6ded2fcf11a49d3da9, for GNU/Linux 3.7.0, stripped

I think it hard to guess from the VM the host architecture due to isolation, I also had a quick look but the only thing that comes close would be lscpu which says it's an aarch64 with only 64-bit support

debian@debian:~$ lscpu
Architecture:           aarch64
  CPU op-mode(s):       64-bit
  Byte Order:           Little Endian
CPU(s):                 6
  On-line CPU(s) list:  0-5
Vendor ID:              0x00
  Model name:           -
    Model:              0
    Thread(s) per core: 1
    Core(s) per socket: 6
    Socket(s):          1
    Stepping:           0x0
    BogoMIPS:           48.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid a
                        simdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat i
                        lrcpc flagm sb paca pacg dcpodp flagm2 frint
NUMA:
  NUMA node(s):         1
  NUMA node0 CPU(s):    0-5
Vulnerabilities:
  Gather data sampling: Not affected
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Not affected
  Mmio stale data:      Not affected
  Retbleed:             Not affected
  Spec rstack overflow: Not affected
  Spec store bypass:    Vulnerable
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Not affected
  Srbds:                Not affected
  Tsx async abort:      Not affected

[gitbls]

Thanks for that. I have an idea of how to fix. I'll put something together for you to try.

[gitbls]

I believe I have corrected some issues that will improve this, but before I give it to you to try, I'd like a preliminary confirmation that I'm on the correct path.

Would you please do the following:

• Reboot the system and DON'T update binfmt

• Provide the output from:

  • ls -l /usr/bin/qemu*
  • ls -l /proc/sys/fs/binfmt_misc

  Thx!

[gitbls]

Moving ahead anyhow. Please try the sdm-cparse in the attached zip file:

• Unzip it
• cd /usr/local/sdm
• sudo mv sdm-cparse sdm-save-cparse
• sudo cp /path/to/new/sdm-cparse sdm-cparse

• sdm-cparse.zip

Yes, I know this can be done with a github branch

[ljames8]

Regarding your previous message here are the outputs:

debian@debian:/media/share$ ls -l /usr/bin/qemu*
-rwxr-xr-x 1 root root 10326544 Feb  6 09:38 /usr/bin/qemu-aarch64_be-static
-rwxr-xr-x 1 root root 10326544 Feb  6 09:38 /usr/bin/qemu-aarch64-static
-rwxr-xr-x 1 root root  7834768 Feb  6 09:38 /usr/bin/qemu-alpha-static
-rwxr-xr-x 1 root root  8950240 Feb  6 09:38 /usr/bin/qemu-armeb-static
-rwxr-xr-x 1 root root  8950240 Feb  6 09:38 /usr/bin/qemu-arm-static
-rwxr-xr-x 1 root root  7834544 Feb  6 09:38 /usr/bin/qemu-cris-static
-rwxr-xr-x 1 root root  9932000 Feb  6 09:38 /usr/bin/qemu-hexagon-static
-rwxr-xr-x 1 root root  7900304 Feb  6 09:38 /usr/bin/qemu-hppa-static
-rwxr-xr-x 1 root root  8461840 Feb  6 09:38 /usr/bin/qemu-i386-static
-rwxr-xr-x 1 root root  7834752 Feb  6 09:38 /usr/bin/qemu-loongarch64-static
-rwxr-xr-x 1 root root  8097040 Feb  6 09:38 /usr/bin/qemu-m68k-static
-rwxr-xr-x 1 root root  7837024 Feb  6 09:38 /usr/bin/qemu-microblazeel-static
-rwxr-xr-x 1 root root  7837024 Feb  6 09:38 /usr/bin/qemu-microblaze-static
-rwxr-xr-x 1 root root  8752736 Feb  6 09:38 /usr/bin/qemu-mips64el-static
-rwxr-xr-x 1 root root  8752736 Feb  6 09:38 /usr/bin/qemu-mips64-static
-rwxr-xr-x 1 root root  8687424 Feb  6 09:38 /usr/bin/qemu-mipsel-static
-rwxr-xr-x 1 root root  8752736 Feb  6 09:38 /usr/bin/qemu-mipsn32el-static
-rwxr-xr-x 1 root root  8752736 Feb  6 09:38 /usr/bin/qemu-mipsn32-static
-rwxr-xr-x 1 root root  8687424 Feb  6 09:38 /usr/bin/qemu-mips-static
-rwxr-xr-x 1 root root  7837584 Feb  6 09:38 /usr/bin/qemu-nios2-static
-rwxr-xr-x 1 root root  7834672 Feb  6 09:38 /usr/bin/qemu-or1k-static
-rwxr-xr-x 1 root root  8812368 Feb  6 09:38 /usr/bin/qemu-ppc64le-static
-rwxr-xr-x 1 root root  8812368 Feb  6 09:38 /usr/bin/qemu-ppc64-static
-rwxr-xr-x 1 root root  8676048 Feb  6 09:38 /usr/bin/qemu-ppc-static
-rwxr-xr-x 1 root root  8889152 Feb  6 09:38 /usr/bin/qemu-riscv32-static
-rwxr-xr-x 1 root root  8889152 Feb  6 09:38 /usr/bin/qemu-riscv64-static
-rwxr-xr-x 1 root root  8240080 Feb  6 09:38 /usr/bin/qemu-s390x-static
-rwxr-xr-x 1 root root  7900544 Feb  6 09:38 /usr/bin/qemu-sh4eb-static
-rwxr-xr-x 1 root root  7900544 Feb  6 09:38 /usr/bin/qemu-sh4-static
-rwxr-xr-x 1 root root  8034320 Feb  6 09:38 /usr/bin/qemu-sparc32plus-static
-rwxr-xr-x 1 root root  8034320 Feb  6 09:38 /usr/bin/qemu-sparc64-static
-rwxr-xr-x 1 root root  7968784 Feb  6 09:38 /usr/bin/qemu-sparc-static
-rwxr-xr-x 1 root root  8396080 Feb  6 09:38 /usr/bin/qemu-x86_64-static
-rwxr-xr-x 1 root root 10712112 Feb  6 09:38 /usr/bin/qemu-xtensaeb-static
-rwxr-xr-x 1 root root 10822608 Feb  6 09:38 /usr/bin/qemu-xtensa-static

debian@debian:/media/share$ ls -l /proc/sys/fs/binfmt_misc
total 0
-rw-r--r-- 1 root root 0 Feb 21 05:39 python3.11
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-alpha
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-armeb
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-cris
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-hexagon
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-hppa
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-i386
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-loongarch64
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-m68k
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-microblaze
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-mips
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-mips64
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-mips64el
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-mipsel
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-mipsn32
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-mipsn32el
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-ppc
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-ppc64
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-ppc64le
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-riscv32
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-riscv64
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-s390x
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-sh4
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-sh4eb
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-sparc
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-sparc32plus
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-sparc64
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-x86_64
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-xtensa
-rw-r--r-- 1 root root 0 Feb 21 05:39 qemu-xtensaeb
--w------- 1 root root 0 Feb 21 05:39 register
-rw-r--r-- 1 root root 0 Feb 21 05:39 status

[ljames8]

And I just tried to sdm --explore with the zip you provided,

• Without updating binfmt, I still get the error

  
  debian@debian:~$ sudo sdm --explore my-image.img

• Mount IMG 'my-image.img' mount: /dev/loop0 mounted on /mnt/sdm. mount: /dev/loop1 mounted on /mnt/sdm/boot/firmware.
• Enter IMG 'my-image.img' execv(/bin/bash) failed: Exec format error umount: /mnt/sdm/boot/firmware unmounted umount: /mnt/sdm unmounted

• However, after updating binfmt, looks like sudo sdm --chroot --explore my-image.img now works!

  
  debian@debian:~$ sudo sdm --chroot --explore my-image.img

• Mount IMG 'my-image.img' mount: /dev/loop0 mounted on /mnt/sdm. mount: /dev/loop1 mounted on /mnt/sdm/boot/firmware. % sdm will use chroot per --chroot on this 'ARM aarch64' host
• Enter IMG 'my-image.img' root@debian:/# root@debian:/# root@debian:/# root@debian:/# exit umount: /mnt/sdm/boot/firmware unmounted umount: /mnt/sdm unmounted

  Here is my sdm directory fyi
  ```bash
  debian@debian:~$ ls -l /usr/local/sdm/
  total 296
  drwxr-xr-x 2 root root  4096 Feb 16 07:13 1piboot
  drwxr-xr-x 2 root root  4096 Feb 16 07:12 local-plugins
  drwxr-xr-x 2 root root  4096 Feb 16 07:13 plugins
  -rwxr-xr-x 1 root root 35613 Feb 16 08:32 sdm
  -rw-r--r-- 1 root root   545 Feb 16 07:13 sdm-apps-example
  -rwxr-xr-x 1 root root   660 Feb 16 07:13 sdm-apt
  -rwxr-xr-x 1 root root  1485 Feb 16 07:13 sdm-apt-cacher
  -rwxr-xr-x 1 root root 19427 Feb 16 07:13 sdm-cmdsubs
  -rwxr-xr-x 1 root root 51888 Feb 21 05:44 sdm-cparse
  -rwxr-xr-x 1 root root 39319 Feb 16 07:13 sdm-cportal
  -rwxr-xr-x 1 root root 16232 Feb 16 07:13 sdm-cryptconfig
  -rwxr-xr-x 1 root root  4496 Feb 16 07:13 sdmcryptfs
  -rwxr-xr-x 1 root root  1295 Feb 16 07:13 sdm-customphase
  -rwxr-xr-x 1 root root  7110 Feb 16 07:13 sdm-firstboot
  -rwxr-xr-x 1 root root  7650 Feb 16 07:13 sdm-gburn
  -rwxr-xr-x 1 root root   142 Feb 16 07:13 sdm-logmsg
  -rwxr-xr-x 1 root root   696 Feb 16 07:12 sdm-phase0
  -rwxr-xr-x 1 root root  7105 Feb 16 08:47 sdm-phase1
  -rwxr-xr-x 1 root root  1312 Feb 16 07:13 sdm-readparams
  -rwxr-xr-x 1 root root  2427 Feb 16 07:13 sdm-rpcsubs
  -rwxr-xr-x 1 root root 51053 Feb 16 08:53 sdm-save-cparse
  -rw-r--r-- 1 root root   929 Feb 16 07:13 sdm-xapps-example

[gitbls]

Good news! For completeness, what is the result if you rerun the command WITHOUT the --chroot (and binfmt updated)?

[ljames8]

Still the same as in my first post:

debian@debian:~$ sudo sdm --explore my-image.img
* Mount IMG 'my-image.img'
mount: /dev/loop0 mounted on /mnt/sdm.
mount: /dev/loop1 mounted on /mnt/sdm/boot/firmware.
* Enter IMG 'my-image.img'
root@sdm:/#
exit
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted

[gitbls]

So as I understand it now, both systemd-nspawn and chroot work now, as long as binfmt is updated? Good news!

I will have an updated version for you that does the binfmt update automatically and hopefully it will all work as expected. 🤞

[gitbls]

I believe that this version does automatic binfmt setting. Could you give it a try?

sdm-cparse.zip

Thx!

[ljames8]

Hi, just tried it, but it does not work (same errors as before) as is.

debian@debian:~$ sudo sdm --explore my-image.img
[sudo] password for debian:
* Mount IMG 'my-image.img'
mount: /dev/loop0 mounted on /mnt/sdm.
mount: /dev/loop1 mounted on /mnt/sdm/boot/firmware.
* Enter IMG 'my-image.img'
execv(/bin/bash) failed: No such file or directory
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted

It does not seems to do something special, whereas with the --chroot option I get

debian@debian:~$ sudo sdm --chroot --explore my-image.img
* Mount IMG 'my-image.img'
mount: /dev/loop0 mounted on /mnt/sdm.
mount: /dev/loop1 mounted on /mnt/sdm/boot/firmware.
% Add binfmt for 'arm' architecture
% sdm will use chroot per --chroot on this 64-bit ARM aarch64 host
* Enter IMG 'my-image.img'
chroot: failed to run command ‘/bin/bash’: No such file or directory
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted

The register part seems to happen but does not seem to succeed. I noticed the diff between what you added and the string I use when manually updating binfmt:

<       bfs=":qemu-arm:M:0:\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:qemu-arm-static:"
---
>       bfs=":qemu-arm:M:0:\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:F"

Not sure what this diff means, but when applying my version on sdm-cparse, it does seem to work

debian@debian:~$ sudo sdm --chroot --explore my-image.img

* Mount IMG 'my-image.img'
mount: /dev/loop0 mounted on /mnt/sdm.
mount: /dev/loop1 mounted on /mnt/sdm/boot/firmware.
% Add binfmt for 'arm' architecture
% sdm will use chroot per --chroot on this 64-bit ARM aarch64 host
* Enter IMG 'my-image.img'
root@debian:/#
exit
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted
debian@debian:~$

[gitbls]

Huh. Could you try the following:

• Edit the 'bfs' string to put the F back in
• Reboot
• Try both without chroot and then with
• If that fails, then put the /usr/bin in, and retry the reboot/test again Thx!

OK, according to https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html the filename should be a full path, so added that. Also added the F flag back, so don't need you to test further until I push out the next release.

[ljames8]

Ok, I tested without chroot, it never updates binfmt and does not work. with chroot, I confirm the only string that seems to work is with an absolute path and F flag

[gitbls]

Thx for confirming the binfmt string contents.

I re-imagined 🤪the initialization yesterday with that exact binfmt contents and explicit handling of arm64 host without 32-bit mode, and I believe this one should do it.

Please test both with --chroot and without, rebooting in between. sdm will notify a) if qemu is going to be used, and b) if binfmt is updated.

sdm-cparse.zip

Thx!

[gitbls]

I've checked in V11.5 that has this incorporated. Please test V11.5 and LMK how it goes. Thx!

[gitbls]

Verified by @garshythoel over in https://github.com/gitbls/sdm/issues/187

[ljames8]

Looks great, thanks a lot

[gitbls]

Great! And thank you for helping me sort this out!

[gitbls]

Closing as resolved.