systemd-vmspawn boot of Arch Linux image hangs at "Switch Root" but mkosi qemu works

niklas88 commented 3 months ago

systemd version the issue has been seen with

256

Used distribution

Arch Linux

Linux kernel version used

6.9.5-arch1-1

CPU architectures issue was seen on

x86_64

Component

systemd-vmspawn

Expected behaviour you didn't see

Got systemd-256 in Arch Linux and want to test mkosi + systemd-vmspawn using the "Example 1" from its man page. The image build works but then the boot inside systemd-vmspawn hangs. So I thought maybe the image is wrong and tried with mkosi --output image.raw qemu but that works.

Unexpected behaviour you saw

This first boots normally but then hangs indefinitely at:

[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create Static Device Nodes in /dev gracefully.
[  OK  ] Stopped Create List of Static Device Nodes.
[  OK  ] Stopped Load udev Rules from Credentials.
[  OK  ] Finished Cleanup udev Database.
[  OK  ] Reached target Switch Root.
         Starting Switch Root...

Waited several minutes then killed it with ctrl+]*3

Steps to reproduce the problem

I built an Arch Linux image using mkosi and the exact command from the systemd-vmspawn Example 1

$ mkosi --version
mkosi 23.1
$ systemd-vmspawn --version
systemd 256 (256.1-1-arch-g34ba18b^)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT +LIBARCHIVE
$ mkosi -d arch -p systemd -p linux,base,systemd --autologin -o image.raw -f build
$ systemd-vmspawn --image=image.raw
…
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create Static Device Nodes in /dev gracefully.
[  OK  ] Stopped Create List of Static Device Nodes.
[  OK  ] Stopped Load udev Rules from Credentials.
[  OK  ] Finished Cleanup udev Database.
[  OK  ] Reached target Switch Root.
         Starting Switch Root...

I first though the image is bad so tried booting it with mkosi's QEMU support

$ mkosi --output=image.raw qemu
…
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.

Arch Linux 6.9.5-arch1-1 (ttyS0)

archlinux login: root (automatic login)

[root@archlinux ~]#

I also tried `sudo systemd-vmspawn --image=image.raw`` with the same behavior.

Additional program output to the terminal or log subsystem illustrating the issue

No response

niklas88 commented 3 months ago

I think this might be related to this mkosi issue. There Debian didn't boot with virtio-scsi-pci hanging at gpt-auto-root. I also tried with a mkosi built Fedora image and that actually hangs at gpt-auto-root though weirdly enough this one doesn't boot in mkosi … qemu either. Also I tested this on another Arch Linux system with the same results.

For Fedora I used the followign mkosi command:

mkosi -d fedora -p kernel --autologin -o fedora.raw -f build

niklas88 commented 3 months ago

Ok, now I'm confused. With Fedora I was trying to debug and used the below modfied mkosi command to get an unlocked root account so the emergency shell would work but with that it just booted all the way to the shell:

mkosi -d fedora -p kernel --autologin -o fedora --root-password='hashed:' -f build
systemd-vmspawn -i fedora.raw

The same didn't help with Arch Linux though.

DaanDeMeyer commented 2 months ago

This should be fixed by #33697 please let us know if you can still see the issue with vmspawn from git main

systemd / systemd