No keyboard input after building with dracut and systemd 256.1 or 256.2 (at least on Arch Linux)

samy-mahmoudi commented 4 months ago

ZFSBootMenu build source

Local build, dracut

ZFSBootMenu version

2.3.0

Boot environment distribution

Arch Linux

Problem description

No keyboard input when being prompted for my passphrase.

With my key included (install_items+=" PATH_TO_MY_KEY ") for debugging purposes and zbm.show: no keyboard input while on the menu.

Possible, related discussion: https://github.com/zbm-dev/zfsbootmenu/discussions/619

Steps to reproduce

Rollback to a ZFS snapshot where the version of systemd, systemd-libs, systemd-sysvcompat and lib32-systemd is 255.7-1.
Upgrade all the packages but the four aforementioned packages.
Generate EFI bundle or components.
Reboot.
Observe working keyboard inputs.
Upgrade the four aforementioned packages (to date, to version 256.1-1) and only those packages.
Generate EFI bundle or components.
Reboot.
Observe broken keyboard inputs.

By default of a relevant ZFS snapshot, I believe the four packages can be downgraded to 255.7-1 at the OS level.

Note

Building with mkinitcpio and systemd 256.1-1 produces working EFI bundles and components.

zdykstra commented 4 months ago

This is not likely a systemd issue, since that Dracut module is explicitly blacklisted in ZFSBootMenu. When you roll back to a previous snapshot, does your kernel also change? If so, what is the kernel in the working snapshot and what is the kernel in the non-working snapshot?

samy-mahmoudi commented 4 months ago

Here are the snapshots I have around the breaking point:

Date: 2024-06-15 Working state: OK Kernel (LTS): 6.6.32-1 Systemd: 255.7-1 ZFS: 2.2.4-1 Dracut: 101

Date: 2024-06-16 Working state: OK Kernel (LTS): 6.6.33-1 Systemd: 255.7-1 ZFS: 2.2.4-1 Dracut: 102-1

Date: 2024-06-20 Working state: Broken (No keyboard input) Kernel (LTS): 6.6.34-1 Systemd: 256.1-1 ZFS: 2.2.4-1 Dracut: 102-1

When I determined the working states by creating clones out of the three snapshots, then booting the environments and eventually building ZFSBootmenu, I realized that between the last working state and the first broken state, both the kernel and systemd had their version changed. So I decided to upgrade the kernel with its headers only at first to exclude a kernel issue, which eventually led to me to the following states:

Date: 2024-06-29 Working state: OK Kernel (LTS): 6.6.36-1 Systemd: 255.7-1 ZFS: 2.2.4-1 Dracut: 102-1

Date: 2024-06-29 Working state: Broken (No keyboard input) Kernel (LTS): 6.6.36-1 Systemd: 256.1-1 ZFS: 2.2.4-1 Dracut: 102-1

classabbyamp commented 4 months ago

can you diff the contents of the EFIs? you can extract with lsinitrd

also, what's the hardware like? I suspect a udev change

samy-mahmoudi commented 4 months ago

Hi again all,

can you diff the contents of the EFIs? you can extract with lsinitrd

I certainly can.

I used a couple of sed commands to filter out the noise induced by the differences in date and time:

[0]samy@x270:/home/samy$ lsinitrd /efi/EFI/arch/zfsbootmenu/initramfs-bootmenu.img-Jun16-kern6.6.38 \
  | sed 's/^.*root.*root.*..:.. //' \
  >Jun16-kern6.6.38

[0]samy@x270:/home/samy$ lsinitrd /efi/EFI/arch/zfsbootmenu/initramfs-bootmenu.img-Jun20-kern6.6.38 \
  | sed 's/^.*root.*root.*..:.. //' \
  >Jun20-kern6.6.38

[0]samy@x270:/home/samy$ diff -u Jun16-kern6.6.38 Jun20-kern6.6.38
--- Jun16-kern6.6.38        2024-07-11 01:02:12.737687529 -0400
+++ Jun20-kern6.6.38        2024-07-11 01:01:51.254354706 -0400
@@ -1,4 +1,4 @@
-Image: /efi/EFI/arch/zfsbootmenu/initramfs-bootmenu.img-Jun16-kern6.6.38: 81M
+Image: /efi/EFI/arch/zfsbootmenu/initramfs-bootmenu.img-Jun20-kern6.6.38: 81M
 ========================================================================
 Early CPIO image
 ========================================================================
@@ -504,18 +504,12 @@
 usr/lib/libkeyutils.so -> libkeyutils.so.1
 usr/lib/libkeyutils.so.1 -> libkeyutils.so.1.10
 -rwxr-xr-x   1 root     root        22400 Apr 27  2023 usr/lib/libkeyutils.so.1.10
-usr/lib/libkmod.so -> libkmod.so.2.4.2
-usr/lib/libkmod.so.2 -> libkmod.so.2.4.2
-usr/lib/libkmod.so.2.4.2
 usr/lib/libkrb5.so -> libkrb5.so.3.3
 usr/lib/libkrb5.so.3 -> libkrb5.so.3.3
 -rwxr-xr-x   1 root     root       882664 Dec 24  2023 usr/lib/libkrb5.so.3.3
 usr/lib/libkrb5support.so -> libkrb5support.so.0.1
 usr/lib/libkrb5support.so.0 -> libkrb5support.so.0.1
 -rwxr-xr-x   1 root     root        55472 Dec 24  2023 usr/lib/libkrb5support.so.0.1
-usr/lib/liblz4.so -> liblz4.so.1
-usr/lib/liblz4.so.1 -> liblz4.so.1.9.4
-usr/lib/liblz4.so.1.9.4
 usr/lib/liblzma.so -> liblzma.so.5.6.2
 usr/lib/liblzma.so.5 -> liblzma.so.5.6.2
 usr/lib/liblzma.so.5.6.2
@@ -577,14 +571,14 @@
 usr/lib/libsmartcols.so.1 -> libsmartcols.so.1.1.0
 usr/lib/libsmartcols.so.1.1.0
 usr/lib/libsystemd.so -> libsystemd.so.0
-usr/lib/libsystemd.so.0 -> libsystemd.so.0.38.0
-usr/lib/libsystemd.so.0.38.0
+usr/lib/libsystemd.so.0 -> libsystemd.so.0.39.0
+usr/lib/libsystemd.so.0.39.0
 usr/lib/libtirpc.so -> libtirpc.so.3.0.0
 usr/lib/libtirpc.so.3 -> libtirpc.so.3.0.0
 -rwxr-xr-x   1 root     root       186800 Oct  7  2023 usr/lib/libtirpc.so.3.0.0
 usr/lib/libudev.so -> libudev.so.1
-usr/lib/libudev.so.1 -> libudev.so.1.7.8
-usr/lib/libudev.so.1.7.8
+usr/lib/libudev.so.1 -> libudev.so.1.7.9
+usr/lib/libudev.so.1.7.9
 usr/lib/libuuid.so -> libuuid.so.1
 usr/lib/libuuid.so.1 -> libuuid.so.1.3.0
 usr/lib/libuuid.so.1.3.0
@@ -2287,7 +2281,7 @@
 usr/lib/ossl-modules/legacy.so
 usr/lib/profiling-lib.sh
 usr/lib/systemd
-usr/lib/systemd/libsystemd-shared-255.7-1.so
+usr/lib/systemd/libsystemd-shared-256.1-1.so
 usr/lib/systemd/network
 usr/lib/systemd/network/80-6rd-tunnel.network
 usr/lib/systemd/network/80-container-host0.network

also, what's the hardware like? I suspect a udev change

The machine is a Lenovo ThinkPad x270 used with its integrated keyboard, which does not have any LED on the CapsLock key (see below).

I have tried to reproduce the issue with an external USB keyboard plugged. Pretty much the same result (no input), with one notable difference: CapsLocks keystrokes do not toggle the CapsLock LED that is present on the external USB keyboard.

samy-mahmoudi commented 4 months ago

Some testing suggests this bug *doesn't* prevent boot,
at least of a simple default Fedora install in a VM;
patching ProtectSystem to default to off for initramfses
doesn't get the system booting again, but patching dracut
to pull in the kmod library does get it booting again.

Surely the inability to write hooks must have some fairly
significant consequences in some cases, though.

Originally posted by @AdamWill in https://github.com/systemd/systemd/issues/32511#issuecomment-2080144756

samy-mahmoudi commented 4 months ago

Note:

Although [dracut-ng/dracut-ng@a45048b] has landed on Arch, I have nonetheless tried to install_items+=" /etc/systemd/system.conf " with the default:

#ProtectSystem=auto

changed to:

ProtectSystem=no

I could then see /etc/systemd/system.conf in the relevant output of lsinitrd, without any more success when booting the ZFSBootMenu RAM disk.

samy-mahmoudi commented 4 months ago

A static sequence of lines:

install_items+=" /usr/lib/libkmod.so "
install_items+=" /usr/lib/libkmod.so.2 "
install_items+=" /usr/lib/libkmod.so.2.4.2 "

in /etc/zfsbootmenu/dracut.conf.d/workaround_lack_of_libkmod.conf works around the issue.

I will now try to figure out how and where (ZFSBootMenu, dracut, systemd) to fix the issue fully. According to the following, annotated references, my guess is that a full fix rather lies between ZFSBootMenu and dracut-ng (in the large sense).

References:

https://github.com/systemd/systemd/pull/31131: origin of this issue, presumably
https://github.com/dracut-ng/dracut-ng/pull/118: attempt to mirror the changes originating from systemd into dracut-ng
https://github.com/systemd/systemd/issues/32508: realization that the two previous moves can potentially cause a regression on, quote unquote, exotic storage, like they actually do in dracut-based ZFSBootMenu RAM disks.

classabbyamp commented 4 months ago

this is probably dracut assuming there will be a full set of systemd stuff in the initramfs. zfsbootmenu blocklists everything systemd it can because it's incompatible with how ZBM works. thus, when systemd-udev is added, it brings in libsystemd (unavoidable and does not cause compatibility issues), it's missing the libkmod that whatever adds udev expected to be added by the thing that added the rest of systemd.

LaszloGombos commented 4 months ago

There is some related work upstream https://github.com/dracut-ng/dracut-ng/pull/507 and some related Gentoo discussion - https://bugs.gentoo.org/935548

samy-mahmoudi commented 4 months ago

Hi,

Thank you all, especially @zdykstra, @classabbyamp, @LaszloGombos and @puleglot.

I will remove /etc/zfsbootmenu/dracut.conf.d/workaround_lack_of_libkmod.conf and port the changes of https://github.com/dracut-ng/dracut-ng/commit/ef0972fe5349bdf6e821cb79a47cf412caf99059 back into my local installation as a test prior to the relevant, upcoming release of dracut-ng.

Just to confirm the changes will fully fix the issue seamlessly from the ZFSBootMenu perspective.

samy-mahmoudi commented 3 months ago

I can confirm the changes of dracut-ng/dracut-ng#507 ported back to dracut-103 (resp. dracut-102) fix the keyboard input.

Let us close this issue accordingly.

zbm-dev / zfsbootmenu