Ubuntu ISO Boot: Failed to start Switch Root (UEFI)

jloeser commented 3 weeks ago

Problem description

I try to build a Ubuntu 22.04 ISO which can be booted directly from ISO and via network via dracut-live.

A built Ubuntu 22.04 ISO fails during boot (via ISO and network) with:

[FAILED] Failed to start Switch Root.
See 'systemctl status initrd-switch-root.service' for details.
Warning: /dev/root does not exist

Expected behaviour

Successful boot.

Steps to reproduce the behaviour

Install KIWI 10.1.16 via pipx on Debian 12
Get https://github.com/OSInside/kiwi-descriptions/commit/d320127790267cdef92536092c330d6f3103a560
ubuntu/x86_64/ubuntu-jammy/config.xml (basically a stripped version for ISO only build and added dracut-live package):

<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="7.6" name="Ubuntu-22.04_appliance">
    <description type="system">
        <author>Marcus Schaefer</author>
        <contact>ms@suse.com</contact>
        <specification>Image description for Ubuntu 22.04</specification>
    </description>
    <profiles>
        <profile name="Live" description="Live image" import="true"/>
    </profiles>
    <preferences>
        <version>1.16.4</version>
        <packagemanager>apt</packagemanager>
        <bootsplash-theme>sabily</bootsplash-theme>
        <bootloader-theme>ubuntu-mate</bootloader-theme>
        <locale>en_US</locale>
        <keytable>us</keytable>
        <timezone>UTC</timezone>
        <rpm-check-signatures>false</rpm-check-signatures>
    </preferences>
    <preferences profiles="Live">
        <type image="iso" flags="dmsquash" hybridpersistent_filesystem="ext4" hybridpersistent="true" firmware="uefi"/>
    </preferences>
    <users>
        <user password="$1$wYJUgpM5$RXMMeASDc035eX.NbYWFl0" home="/root" name="root" groups="root"/>
    </users>
    <repository type="apt-deb" repository_gpgcheck="false" package_gpgcheck="false" imageinclude="true">
        <source path="obs://Virtualization:Appliances:Builder:Deprecated/xUbuntu_22.04"/>
    </repository>
    <repository type="apt-deb" alias="Jammy-security" distribution="jammy-security" components="main multiverse restricted universe" repository_gpgcheck="false">
        <source path="http://security.ubuntu.com/ubuntu"/>
    </repository>
    <repository type="apt-deb" alias="Jammy-updates" distribution="jammy-updates" components="main multiverse restricted universe" repository_gpgcheck="false">
        <source path="http://us.archive.ubuntu.com/ubuntu/"/>
    </repository>
    <repository type="apt-deb" alias="Jammy" distribution="jammy" components="main multiverse restricted universe" repository_gpgcheck="false">
        <source path="http://us.archive.ubuntu.com/ubuntu/"/>
    </repository>
    <packages type="image">
        <package name="libpam-runtime"/>
        <package name="grub2-themes-ubuntu-mate"/>
        <package name="plymouth-theme-sabily"/>
        <package name="plymouth"/>
        <package name="grub-efi-amd64"/>
        <package name="grub-common"/>
        <package name="grub2-common"/>
        <package name="grub-pc-bin"/>
        <package name="linux-generic"/>
        <package name="isolinux"/>
        <package name="syslinux"/>
        <package name="syslinux-common"/>
        <package name="systemd"/>
        <package name="dracut"/>
        <package name="init"/>
        <package name="gnupg"/>
        <package name="iproute2"/>
        <package name="iptables"/>
        <package name="iputils-ping"/>
        <package name="ifupdown"/>
        <package name="isc-dhcp-client"/>
        <package name="netbase"/>
        <package name="dbus"/>
        <package name="xz-utils"/>
        <package name="btrfs-progs"/>
        <package name="shim"/>
        <package name="shim-signed"/>
    </packages>
    <packages type="iso">
    <package name="dracut-live"/>
        <package name="dracut-kiwi-live"/>
    </packages>
    <packages type="bootstrap">
        <package name="usrmerge"/>
    </packages>
</image>

Build it, boot it (from ISO)

Investigation

Something seems wrong with service dependencies in initrd procuded by dracut.

The following manual steps in the dracut emergency shell make the system boot up successfully:

:/root# systemctl start sysroot.mount
[   13.523613] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
:/root# systemctl start initrd-switch-root.service

So unit file-wise, everything needed seems to be available in initrd.

Unit files:

:/root# systemctl cat sysroot.mount
# /run/systemd/generator.early/sysroot.mount
[Unit]
Before=initrd-root-fs.target
[Mount]
Where=/sysroot
What=/dev/mapper/live-rw
:/root# systemctl cat initrd-switch-root.service
# /lib/systemd/system/initrd-switch-root.service
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Switch Root
DefaultDependencies=no
AssertPathExists=/etc/initrd-release
OnFailure=emergency.target
OnFailureJobMode=replace-irreversibly
AllowIsolate=yes

[Service]
Type=oneshot
ExecStart=systemctl --no-block switch-root /sysroot

I came up with this awkward workaround which produces a bootable ISO for me:

Add a breakpoint (PDB) just before dracut call (https://github.com/OSInside/kiwi/blob/c98ec19142586e3ebf9350f2544629372fecbff0/kiwi/boot/image/dracut.py#L212)
Start build kiwi-ng system build --description ubuntu/x86_64/ubuntu-jammy/, wait for breakpoint

Apply the following patch to the initrd-switch-root.service file in the image build directory:

--- img/build/image-root/lib/systemd/system/initrd-switch-root.service  2022-03-11 12:48:49.000000000 +0000
+++ /root/initrd-switch-root.service    2024-10-29 08:37:22.551246985 +0000
@@ -14,6 +14,8 @@
OnFailure=emergency.target
OnFailureJobMode=replace-irreversibly
AllowIsolate=yes
+Requires=sysroot.mount
+After=sysroot.mount

[Service]
Type=oneshot

Continue the build (c in PDB prompt)
ISO is booting up successfully

OS and Software information

KIWI version: 10.1.16
Operating system host version: Debian GNU/Linux 12 (bookworm)
Operating system target version: Ubuntu 22.04
Open Build Service version (N/A if not using OBS):
Koji version (N/A if not using Koji):

schaefi commented 3 weeks ago

Hmm, the part which is unclear to me is why you think this is an issue that needs to be fixed in kiwi ?

I assume /lib/systemd/system/initrd-switch-root.service is provided by a package of the Ubuntu system that you are using. The change you added to the service unit looks good to me but does not fall into the responsibility of kiwi.

Instead of the break point in code it should imho also be possible to add the above patch file as overlay file e.g root/sysroot.patch to your kiwi image description and add a line in your config.sh script that does

set -ex

patch -p0 < sysroot.patch
rm /sysroot.patch

This will modify a file in your root tree as sort of a "hot fix" and should have the same effect. Until things will be fixed on Ubuntu side you can maintain the change in your kiwi image description.

I don't see what else we can offer in this regard. In addition please also note that Ubuntu 22.04 is outside of the support matrix for kiwi v10.

@Conan-Kudo Any thoughts you want to add here ?

jloeser commented 3 weeks ago

Hmm, the part which is unclear to me is why you think this is an issue that needs to be fixed in kiwi ?

Oh, you are right! Looks like I did mix up things because I was also playing around with the dracut-kiwi-live package.

Anyway, thanks for the hint with the hot fix in the config.sh - this is definitely a way to go.

Sorry for any circumstances, from my point of view we can close this issue.

OSInside / kiwi