systemd-gpt-auto-generator settings (Dependency failed for Initrd Root Device)

[hv15]

Hi there,

Problem:

I keep experience a delay on system startup due to systemd not being able to load the root device (see journalctl output bellow). To be clear, the system does boot and eventually I reach a TTY to login to. Funnily though, my /boot partition does not get mounted, I suspect this is due to the dependency on the root device.

``` Jun 20 15:12:14 sys1 systemd[1]: dev-gpt\x2dauto\x2droot.device: Job dev-gpt\x2dauto\x2droot.device/start timed out. Jun 20 15:12:14 sys1 systemd[1]: Timed out waiting for device /dev/gpt-auto-root. Jun 20 15:12:14 sys1 systemd[1]: Dependency failed for Initrd Root Device. Jun 20 15:12:14 sys1 systemd[1]: initrd-root-device.target: Job initrd-root-device.target/start failed with result 'dependency'. Jun 20 15:12:14 sys1 systemd[1]: dev-gpt\x2dauto\x2droot.device: Job dev-gpt\x2dauto\x2droot.device/start failed with result 'timeout'. ``` ### System Information and mkinitcpio-systemd-tool Configuration I use Arch Linux (linux-lts kernel). My boot drive has two partitions on it, one for `boot` (EFI vFat) and the other is the root partition, setup as LVM on LUKs. The LVM partition has two volume groups (VGs), one for the root filesystem (called `vg-root`) and one for the home directory (called `vg-home`). I use systemd-boot with the these kernel parameters: ``` ipv6.disable_ipv6=1 vsyscall=emulate intel_idle.max_cstate=1 quiet rw` ``` (notice that I omit `root=`). #### Versions ``` $ systemctl --version systemd 245 (245.6-7-arch) +PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid $ uname -a Linux sys1 5.4.47-1-lts #1 SMP Wed, 17 Jun 2020 19:42:02 +0000 x86_64 GNU/Linux ``` #### systemd services I have the following mkinitcpio-systemd-tool services enabled: ``` $ find /etc/systemd/system -type l -iname 'initrd*' /etc/systemd/system/initrd-debug-shell.service.wants/initrd-debug-progs.service /etc/systemd/system/initrd-root-fs.target.wants/initrd-sysroot-mount.service /etc/systemd/system/sysinit.target.wants/initrd-cryptsetup.path /etc/systemd/system/sysinit.target.wants/initrd-debug-progs.service /etc/systemd/system/sysinit.target.wants/initrd-dropbear.service ``` #### Files `mkinitcpio-systemd-tool/config/crypttab` has only this: ``` cryptlvm UUID=3cdf6a91-1a36-4cdc-8eb4-fbd2d799d8d9 ``` `mkinitcpio-systemd-tool/config/fstab` has only this (where `vg-root` is the name of the LVM VG containing the root filesystem): ``` /dev/mapper/vg-root /sysroot auto x-systemd.device-timeout=9999h 0 1 ``` `crypttab` has: ``` cryptlvm UUID=3cdf6a91-1a36-4cdc-8eb4-fbd2d799d8d9 ``` `fstab` has: ``` # /dev/mapper/vg-root UUID=30803547-45e3-4905-b601-ecb92522c40c / ext4 rw,relatime,data=ordered 0 1 # /dev/mapper/vg-home UUID=76d8381f-fc87-4f22-b32a-6c1873b12ad1 /home ext4 rw,relatime,data=ordered 0 2 UUID=E361-10AD /boot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 2 ``` Is my setup of mkinitcpio-systemd-tool wrong? I followed the 'instructions' as outlined in https://github.com/random-archer/mkinitcpio-systemd-tool/issues/62.

Any suggestions are welcome :smile_cat: !

[Andrei-Pozolotin]

1. please confirm lvm is present in /etc/mkinitcpio.conf

   HOOKS=(base keyboard autodetect modconf block filesystems fsck systemd sd-lvm2 systemd-tool)

2. try to read through more lvm-related wiki cases https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case:-Sysroot-on-LVM https://github.com/random-archer/mkinitcpio-systemd-tool/wiki/Case:-Complete-BTRFS-Setup-(step-by-step)

[hv15]

Hi @Andrei-Pozolotin, oops forgot to include content of mkinitcpio.conf (hooks):

HOOKS=(base keyboard autodetect modconf block filesystems fsck systemd sd-vconsole sd-lvm2 systemd-tool)

[Andrei-Pozolotin]

assume resolved

[dvzrv]

I'm actually running into the same issue on several systems (with a system setup close to the one described above).

When booting without network access (e.g. on a laptop), after typing in the password, a start job for dev-gpt-auto-root.device blocks early userspace for 1:30m. After that boot commences. When booting with network access the message about the service is never shown, but I can see it in the list of units (i.e. systemctl list-units -a) as "starting" when ssh'ing to the machine and using the shell.

Fun fact: My fstab for mkinitcpio-systemd-tool does not have a x-systemd.device-timeout=9999h set, so eventually the Starting Cryptography Setup for xxx... fails. As I have now also added the sd-encrypt hook to my mkinitcpio.conf, I can still ssh to the machine after the timeout is reached, type in my password and the boot goes through instantly (no .device unit blocking). This is of course no improvement (as waiting for the timeout takes even longer), but an interesting observation about this issue.

[dvzrv]

Also, this entire issue does not appear, when I do not use crypttab and fstab in the initrd but rely on kernel commandline parameters.

[Andrei-Pozolotin]

@dvzrv great, since you can actually reproduce this :-)

[dvzrv]

I believe I have figured out what triggers this behavior (in my case), now that I had some time to read up on things.

The systemd-gpt-auto-generator acts according to the "discoverable partitions specification", which in turn makes a few assumptions about the layout of a partition.

My system facilitates btrfs on LUKS in a flat hierarchy. Here the default subvolume (id 5) is not mounted as /, but a specific subvolume below it is used for that. The opened device is named root0 in my case (as I have an automatic way of installing systems using ansible, that may use more than one disk to e.g. form a btrfs raid). This does not seem to go too well with the aforementioned generator which probably tries to use the default subvolume (id 5) as root until timing out (also, its default is to open the device as root).

I have now added systemd.gpt_auto=no to my kernel parameters to disable the generator altogether (as fstab is in use anyways) and the systemd boots without delay.

@hv15 I hope this will also fix your issues (which seem to be of the same nature, but I'm not 100% sure). @Andrei-Pozolotin I guess this is worth documenting, because I believe I am not the only one who is using a flat hierarchy for the btrfs subvolumes.

[ArchangeGabriel]

@dvzrv What is there inside your /etc/mkinitcpio-systemd-tool/config/fstab? In my case I specified the subvolume there (because I also use a flat hierarchy) and it works.

[dvzrv]

@ArchangeGabriel it is only:

/etc/mkinitcpio-systemd-tool/config/fstab:

/dev/mapper/root0      /sysroot         btrfs           noatime,ssd,subvolid=259,subvol=/@,subvol=@     0       1

On that specific machine I only have a subvolume for home otherwise, so that's not important in early userspace (which is why I only specify /sysroot.

That being said, this is my list of subvolumes:

D 256 gen 17832 top level 5 path @home
ID 257 gen 8 top level 5 path @swap
ID 259 gen 17830 top level 5 path @
ID 262 gen 25 top level 259 path @/var/lib/portables
ID 263 gen 26 top level 259 path @/var/lib/machines

[dvzrv]

@ArchangeGabriel did you set your btrfs subvolume used for / as the default (i.e. btrfs subvolume set-default)? I did not and maybe that is the culprit (doing it is also mentioned in the generator's man page).

[ArchangeGabriel]

No, I did not. Over I notice that you specified 3 time the subvol directive, not sure that’s a good thing (I know all three settings are correct, but ykk). My line is just:

/dev/mapper/root    /sysroot    auto    subvol=@,x-systemd.device-timeout=9999h 0   1

[Andrei-Pozolotin]

@dvzrv re: doc: added to wiki home: https://github.com/random-archer/mkinitcpio-systemd-tool/wiki

[dvzrv]

@ArchangeGabriel ah, I believe that it is because your device opens as /dev/mapper/root, mine as /dev/mapper/root0. The generator defaults to using root as device according to their specification. I guess in my case the generator finds a btrfs device through device-mapper, but it is not able to mount it, because it is already mounted via root0 at that point in time.