Pacman hook not triggered on kernel update

[ShellCode33]

Hey it's been a while :)

After a kernel upgrade, I'm unable to boot and get the following error:

Nov 01 14:15:38 laptop bootctl[563]: No ESP found, not initializing random seed.
Nov 01 14:15:38 laptop systemd-pcrphase[564]: Extended PCR index 11 with 'sysinit' (banks sha1, sha256).
Nov 01 14:15:40 laptop apparmor.systemd[357]: Restarting AppArmor
Nov 01 14:15:40 laptop apparmor.systemd[357]: Reloading AppArmor profiles
Nov 01 14:15:40 laptop systemd-fsck[366]: fsck.fat 4.2 (2021-01-31)
Nov 01 14:15:40 laptop systemd-fsck[366]: /dev/sda2: 11 files, 40218/140690 clusters
Nov 01 14:15:40 laptop mount[497]: mount: /efi: unknown filesystem type 'vfat'.
Nov 01 14:15:40 laptop mount[497]:        dmesg(1) may have more information after failed mount system call.

I've had this issue a few times now, I'm not sure where it comes from to be honest, but I thought I would write it here in case you have any clue what's going on.

To fix this error I have to boot into a livecd and run arch-secure-boot generate-efi.

Here are the pacman logs that caused the issue:

Click to expand

``` [2023-10-30T19:16:44+0100] [PACMAN] Running '/usr/bin/pacman -S -y -u --config /etc/pacman.conf --' [2023-10-30T19:16:44+0100] [PACMAN] synchronizing package lists [2023-10-30T19:16:45+0100] [PACMAN] starting full system upgrade [2023-10-30T19:16:56+0100] [ALPM] running '05-snap-pac-pre.hook'... [2023-10-30T19:16:57+0100] [ALPM-SCRIPTLET] ==> root: 574 [2023-10-30T19:16:57+0100] [ALPM] running '60-mkinitcpio-remove.hook'... [2023-10-30T19:16:57+0100] [ALPM] transaction started [2023-10-30T19:16:57+0100] [ALPM] upgraded bash (5.1.016-4 -> 5.2.015-5) [2023-10-30T19:16:57+0100] [ALPM] upgraded containerd (1.7.7-1 -> 1.7.8-1) [2023-10-30T19:16:57+0100] [ALPM] upgraded krb5 (1.20.1-1 -> 1.20.1-2) [2023-10-30T19:16:57+0100] [ALPM] upgraded libnghttp2 (1.57.0-1 -> 1.58.0-1) [2023-10-30T19:16:58+0100] [ALPM] upgraded docker (1:24.0.6-1 -> 1:24.0.7-1) [2023-10-30T19:16:58+0100] [ALPM] upgraded fzf (0.42.0-1 -> 0.43.0-1) [2023-10-30T19:16:58+0100] [ALPM] upgraded gpgme (1.23.0-1 -> 1.23.1-1) [2023-10-30T19:16:58+0100] [ALPM] upgraded libyuv (r2322+3aebf69d-1 -> r2426+464c51a0-1) [2023-10-30T19:16:58+0100] [ALPM] upgraded libavif (1.0.1-3 -> 1.0.1-4) [2023-10-30T19:16:58+0100] [ALPM] upgraded linux-firmware-whence (20230804.7be2766d-2 -> 20230918.3672ccab-1) [2023-10-30T19:16:59+0100] [ALPM] upgraded linux-firmware (20230804.7be2766d-2 -> 20230918.3672ccab-1) [2023-10-30T19:17:00+0100] [ALPM] upgraded shadow (4.14.1-1 -> 4.14.2-1) [2023-10-30T19:17:02+0100] [ALPM] upgraded linux-hardened (6.5.8.hardened1-1 -> 6.5.9.hardened1-1) [2023-10-30T19:17:07+0100] [ALPM] upgraded linux-hardened-headers (6.5.8.hardened1-1 -> 6.5.9.hardened1-1) [2023-10-30T19:17:08+0100] [ALPM] upgraded restic (0.16.0-1 -> 0.16.1-1) [2023-10-30T19:17:08+0100] [ALPM] transaction completed [2023-10-30T19:17:09+0100] [ALPM] running '20-systemd-sysusers.hook'... [2023-10-30T19:17:09+0100] [ALPM] running '30-systemd-daemon-reload.hook'... [2023-10-30T19:17:09+0100] [ALPM] running '30-systemd-tmpfiles.hook'... [2023-10-30T19:17:10+0100] [ALPM] running '30-systemd-udev-reload.hook'... [2023-10-30T19:17:11+0100] [ALPM] running '30-systemd-update.hook'... [2023-10-30T19:17:11+0100] [ALPM] running '60-depmod.hook'... [2023-10-30T19:17:13+0100] [ALPM] running '90-mkinitcpio-install.hook'... [2023-10-30T19:17:13+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-hardened.preset: 'default' [2023-10-30T19:17:13+0100] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf' [2023-10-30T19:17:13+0100] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux-hardened -g /boot/initramfs-linux-hardened.img --microcode /boot/intel-ucode.img [2023-10-30T19:17:13+0100] [ALPM-SCRIPTLET] ==> Starting build: '6.5.9-hardened1-1-hardened' [2023-10-30T19:17:13+0100] [ALPM-SCRIPTLET] -> Running build hook: [base] [2023-10-30T19:17:13+0100] [ALPM-SCRIPTLET] -> Running build hook: [consolefont] [2023-10-30T19:17:13+0100] [ALPM-SCRIPTLET] -> Running build hook: [keymap] [2023-10-30T19:17:14+0100] [ALPM-SCRIPTLET] -> Running build hook: [udev] [2023-10-30T19:17:14+0100] [ALPM-SCRIPTLET] -> Running build hook: [autodetect] [2023-10-30T19:17:14+0100] [ALPM-SCRIPTLET] -> Running build hook: [modconf] [2023-10-30T19:17:14+0100] [ALPM-SCRIPTLET] -> Running build hook: [block] [2023-10-30T19:17:16+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'xhci_pci' [2023-10-30T19:17:17+0100] [ALPM-SCRIPTLET] -> Running build hook: [plymouth] [2023-10-30T19:17:18+0100] [ALPM-SCRIPTLET] -> Running build hook: [encrypt] [2023-10-30T19:17:20+0100] [ALPM-SCRIPTLET] -> Running build hook: [filesystems] [2023-10-30T19:17:20+0100] [ALPM-SCRIPTLET] -> Running build hook: [keyboard] [2023-10-30T19:17:22+0100] [ALPM-SCRIPTLET] ==> Generating module dependencies [2023-10-30T19:17:22+0100] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-hardened.img' [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] ==> Image generation successful [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-hardened.preset: 'fallback' [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf' [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux-hardened -g /boot/initramfs-linux-hardened-fallback.img -S autodetect --microcode /boot/intel-ucode.img [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] ==> Starting build: '6.5.9-hardened1-1-hardened' [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] -> Running build hook: [base] [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] -> Running build hook: [consolefont] [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] -> Running build hook: [keymap] [2023-10-30T19:17:23+0100] [ALPM-SCRIPTLET] -> Running build hook: [udev] [2023-10-30T19:17:24+0100] [ALPM-SCRIPTLET] -> Running build hook: [modconf] [2023-10-30T19:17:24+0100] [ALPM-SCRIPTLET] -> Running build hook: [block] [2023-10-30T19:17:25+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'aic94xx' [2023-10-30T19:17:25+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'bfa' [2023-10-30T19:17:26+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qed' [2023-10-30T19:17:26+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla1280' [2023-10-30T19:17:26+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla2xxx' [2023-10-30T19:17:26+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'wd719x' [2023-10-30T19:17:27+0100] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'xhci_pci' [2023-10-30T19:17:30+0100] [ALPM-SCRIPTLET] -> Running build hook: [plymouth] [2023-10-30T19:17:31+0100] [ALPM-SCRIPTLET] -> Running build hook: [encrypt] [2023-10-30T19:17:33+0100] [ALPM-SCRIPTLET] -> Running build hook: [filesystems] [2023-10-30T19:17:34+0100] [ALPM-SCRIPTLET] -> Running build hook: [keyboard] [2023-10-30T19:17:39+0100] [ALPM-SCRIPTLET] ==> Generating module dependencies [2023-10-30T19:17:40+0100] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-hardened-fallback.img' [2023-10-30T19:17:41+0100] [ALPM-SCRIPTLET] ==> Image generation successful [2023-10-30T19:17:42+0100] [ALPM] running 'gdk-pixbuf-query-loaders.hook'... [2023-10-30T19:17:42+0100] [ALPM] running 'post-20-dash-symlink.hook'... [2023-10-30T19:17:42+0100] [ALPM] running 'texinfo-install.hook'... [2023-10-30T19:17:42+0100] [ALPM] running 'zz-snap-pac-post.hook'... [2023-10-30T19:17:42+0100] [ALPM-SCRIPTLET] ==> root: 575 [2023-10-30T19:17:42+0100] [ALPM] running 'zzz-arch-secure-boot-generate-snapshots.hook'... ```

It seems that for some reason the 95-arch-secure-boot-generate-efi hook is not being triggered. Therefore the signed UKI is not being updated and the system becomes unbootable. I'm not sure why it becomes unbootable though, might be due to systemd, I found the following in my kernel logs:

Nov 01 14:15:37 laptop systemd-modules-load[288]: Failed to find module 'crypto_user'
Nov 01 14:15:37 laptop systemd-modules-load[288]: Failed to find module 'dm-multipath'
Nov 01 14:15:37 laptop systemd-modules-load[288]: Failed to find module 'pkcs8_key_parser'

I'm wondering if Operation = Upgrade in the pacman hook is enough. Here's how dracut-hook from the AUR does it:

[Trigger]
Type = Path
Operation = Install
Operation = Upgrade
Target = usr/lib/modules/*/vmlinuz
Target = usr/lib/dracut/*
Target = usr/lib/systemd/systemd

[Action]
Description = Updating initramfs...
When = PostTransaction
Exec = /usr/share/libalpm/scripts/dracut-install
NeedsTargets

We can see that they use both Operation = Install and Operation = Upgrade, and that they also trigger the hook when systemd is being updated.

21 might be related

Let me know what you think :)

[maximbaz]

Hello again :grin:

First of all, many thanks for the very detailed report!

I think your suspicion is entirely correct, it looks like the missing Operation = Install is indeed to blame.

Re-reading the docs again:

Operation = Install|Upgrade|Remove

Select the type of operation to match targets against. May be specified multiple times. >>>Installations are considered an upgrade if the package >or file< is already present on the system<<< regardless of whether the new package version is actually greater than the currently installed version

If we now look at the package contents of linux-hardened (or any other kernel for that matter), the file is being created in a new versioned directory every time:

usr/lib/modules/6.5.9-hardened1-1-hardened/vmlinuz

So by this logic, kernel upgrade would always trigger Install operation, not Upgrade.

Would you like to do the honors of re-introducing Operation = Install? You deserve the full credit here :slightly_smiling_face:

As we talked in #19, the only downside I think is that the initial installation would fire Error: Secure Boot keys are not generated yet error, but I think it's acceptable price.

[ShellCode33]

Sure I can submit a PR. What about Target = usr/lib/systemd/systemd do you think it's worth adding ?

EDIT: do you want me to create a separate hook for Target = usr/bin/arch-secure-boot so that it can be Operation = Upgrade only and prevent the error you are mentioning ?

[maximbaz]

I'm not sure about usr/lib/systemd/systemd itself, but perhaps usr/lib/systemd/boot/efi/linuxx64.efi.stub, since we use it in generating efi files? I don't know how usr/lib/systemd/systemd itself plays a role in efi files, what changes when that binary gets updated...

I think lets avoid a separate hook for now, keep it simple. If nothing else, it is a message to user that they must do something after installation of arch-secure-boot :sweat_smile: