Regression in 252: Secure boot override does not work with some firmware

[patrakov]

systemd version the issue has been seen with

systemd 252.1-1

Used distribution

Arch Linux

Linux kernel version used

6.0.8-zen1-1-zen

CPU architectures issue was seen on

x86_64

Component

systemd-boot, systemd-stub

Expected behaviour you didn't see

Successful boot, with the unified kernel image signed by my custom key (managed by https://github.com/Foxboron/sbctl)

Unexpected behaviour you saw

The first red error is from systemd-boot itself. It says:

Failed to reconnect handle 453, ignoring: Invalid Parameter

The boot menu is shown correctly after that. There was no such error in 251.7-4.

Upon selecting Arch Linux, there are two errors displayed:

Error loading kernel image: Access Denied

and

Failed to execute Arch Linux (home) (\EFI\ArchLinux\archlinux-linux-zen.efi): Access Denied

Steps to reproduce the problem

The problem surfaced when updating Arch Linux - the system didn't come up successfully after a reboot, while other boot entries worked.

Downgrading all systemd-related packages to 251.7-4 and regenerating the EFI image fixes the problem, upgrading again to 252.1-1 re-introduces it.

This is on a very old motherboard, MSI Z87I. I have not tried reproducing this in qemu.

Additional program output to the terminal or log subsystem illustrating the issue

( 9/13) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux-zen.preset: 'default'
  -> -k /boot/vmlinuz-linux-zen -c /etc/mkinitcpio.conf -U /efi/EFI/ArchLinux/archlinux-linux-zen.efi -g /boot/initramfs-linux-zen.img --splash /usr/share/systemd/bootctl/splash-arch.bmp --microcode /boot/intel-ucode.img
==> Starting build: 6.0.8-zen1-1-zen
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
  -> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
  -> Running build hook: [resume]
  -> Running build hook: [btrfs]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-zen.img
==> Image generation successful
==> Creating UEFI executable: /efi/EFI/ArchLinux/archlinux-linux-zen.efi
  -> Using UEFI stub: /usr/lib/systemd/boot/efi/linuxx64.efi.stub
  -> Using kernel image: /lib/modules/6.0.8-zen1-1-zen/vmlinuz
  -> Using cmdline file: /etc/kernel/cmdline
  -> Using os-release file: /etc/os-release
  -> Using splash image: /usr/share/systemd/bootctl/splash-arch.bmp
  -> Using microcode image: /boot/intel-ucode.img
==> UEFI executable generation successful

...

(13/13) Signing EFI binaries...
Generating EFI bundles....
✓ Signed /efi/EFI/ArchLinux/archlinux-linux-zen.efi
File has already been signed /efi/EFI/BOOT/BOOTX64.EFI
File has already been signed /efi/EFI/systemd/systemd-bootx64.efi
File has already been signed /usr/lib/fwupd/efi/fwupdx64.efi.signed

[medhefgo]

Please check this branch. It adds some debug logging that should help shed some light on things. (You you can put the sd-boot build into /EFI/Linux on your esp and then select the new boot entry. You will also need to build an alternative unified image with the build as it contains some extra logs too.)

[patrakov]

Sure, I will try to find some time on weekends.

[solsticedhiver]

I have opened that bug https://bugs.archlinux.org/task/76518 that seems to be a systemd 252 regression too. Though I have found some people on the forum, talking about a similar problem with 245 version. Link in the bug link above

[medhefgo]

I have opened that bug https://bugs.archlinux.org/task/76518 that seems to be a systemd 252 regression too. Though I have found some people on the forum, talking about a similar problem with 245 version. Link in the bug link above

That's unrelated. Please file a separate issue for that.

[patrakov]

Please check this branch. It adds some debug logging that should help shed some light on things.

Done.

The first message is now:

Failed to reconnect handle 453 (D586E218), ignoring: Invalid Parameter

The other two messages stay the same:

Error loading kernel image: Access Denied Failed to execute Arch Linux (home) (\EFI\ArchLinux\archlinux-linux-zen.efi): Access Denied

Disabling Secure Boot allows the system to boot successfully, let me double-check if it also gets rid of the first error...

[patrakov]

It does not get rid of the first error, but the hex number in parens depends on the BIOS settings. E.g., disabling the "Windows 8/8.1 Feature" changes it to D5859618.

[medhefgo]

I've updated my branch, please try again. The first message is not secure boot related and should not appear anymore.

But I expected some more logs from the stub. Are you sure you've booted an entry that uses the updated stub? I've added some extra logs that should always trigger now to make sure.

[patrakov]

Just a stupid question - is it sufficient to replace only the files in /usr/lib/systemd/boot/efi instead of doing the full systemd install?

  488  cp ./src/build/src/boot/efi/systemd-bootx64.efi /usr/lib/systemd/boot/efi/
  489  cp ./src/build/src/boot/efi/systemd-bootx64.efi /efi/EFI/systemd/systemd-bootx64.efi 
  490  cp ./src/build/src/boot/efi/systemd-bootx64.efi /efi/EFI/BOOT/BOOTX64.EFI 
  491  sbctl sign /efi/EFI/systemd/systemd-bootx64.efi
  492  sbctl sign /efi/EFI/BOOT/BOOTX64.EFI
  493  cp ./src/build/src/boot/efi/*.stub /usr/lib/systemd/boot/efi/
...
  497  strings /usr/lib/systemd/boot/efi/linuxx64.efi.stub | grep systemd-git
  498  mkinitcpio -P
  499  strings /efi/EFI/ArchLinux/archlinux-linux-zen.efi | grep systemd-git   # returns the same
  500  sbctl sign /efi/EFI/ArchLinux/archlinux-linux-zen.efi

[medhefgo]

Yes, that should be enough.

[patrakov]

And then the old one:

Failed to execute Arch Linux (home) (\EFI\ArchLinux\archlinux-linux-zen.efi): Access Denied

[patrakov]

Should I bisect where this failure got introduced?

[medhefgo]

Should I bisect where this failure got introduced?

That won't be necessary.

We have moved to a different way the inner kernel is loaded that relies less on hacks. As part of this process we install a secure boot override (so the kernel does not have to be signed) that does not appear to work with your firmware. This is something I half expected and I am too not sure how to deal with.

Maybe we should fall back to the EFI handover protocol in these cases while it's still supported by us… In the meantime you can fix this by having the kernel signed before it's embedded in the UKI. In fact, it's something that sbctl should really be doing anyways…

Incidentally, has the combination of shim+sd-boot (pre 252) ever worked for you?

[patrakov]

I have never used shim.

[patrakov]

Maybe we should fall back to the EFI handover protocol in these cases while it's still supported by us… In the meantime you can fix this by having the kernel signed before it's embedded in the UKI. In fact, it's something that sbctl should really be doing anyways…

In this case, it would require an ABA interaction between mkinitcpio and sbsign, which is currently impossible:

1. A new linux package is installed, and pacman runs all its hooks
2. sbctl should sigh the vmlinuz file at this point, but currently doesn't know that it should do it.
3. mkinitcpio produces the initrd and combines it with everything else by running objcopy, resulting in the unsigned Unified Kernel Image
4. sbctl should sign the UKI at this point, and does it

[medhefgo]

I updated the branch to contain an alternative way to get the firmware to trust the image. Please give it a try (obviously with an unsigned inner image).

[patrakov]

It boots, with these messages:

stub_volume_read_section load_image_by_firmware_volume: Success EFI stub: ERROR: Command line is too long: truncated to 214 bytes

The kernel command line is:

root=/dev/mapper/root rw cryptdevice=/dev/disk/by-uuid/c0e02897-a9a5-4470-a32d-70b8c4f50948:root rootdelay=60 zswap.enabled=1 usbcore.autosuspend=0 log_buf_len=1M intel_iommu=on,igfx_off iommu=pt bgrt_disable quiet

And it is correct (matches /etc/kernel/cmdline). So the truncation warning is bogus.

[patrakov]

Including a signed kernel into the UKI also works fine with systemd 252.1-1.

[patrakov]

But is it actually safe to sign the bare linux kernel, if the goal is to prevent any initrds from unknown origin from being loaded? So far, before this bug surfaced, I achieved this goal by not signing anything except UKIs, and removing the Microsoft certificate (so that the physical attacker, aka evil maid, has no way to load shim and grub).

[inthreedee]

@medhefgo

Incidentally, has the combination of shim+sd-boot (pre 252) ever worked for you?

Just popping in to say that, yes, this was working fine prior to this update on Arch by following the wiki's shim instructions.

FYI, anyone trying to install and test the current release of shim may run into this issue though: https://github.com/rhboot/shim/issues/490

[patrakov]

@inthreedee Do you also have a system affected by this "Access Denied" bug (i.e. Secure Boot not booting unless you sign both the kernel and the UKI containing the said kernel)?

[inthreedee]

@patrakov Yes. I've already downgraded systemd so I can't confirm the fix you're talking about though. Can a pacman hook be used to sign the kernel at the stage you're talking about?

[patrakov]

Yes, that's what I did. Add a file, /etc/pacman.d/hooks/89-sbctl.hook:

[Trigger]
Type = Path
Operation = Install
Operation = Upgrade
Operation = Remove
Target = usr/lib/modules/*/vmlinuz

[Action]
Description = Signing Linux kernels...
When = PostTransaction
Exec = /bin/sh -c 'for file in /usr/lib/modules/*/vmlinuz ; do /usr/bin/sbctl sign "$file" ; done'

[patrakov]

@inthreedee it would also be nice if you name the affected system board if it is a desktop board, or provide the laptop name if it is a laptop.

[inthreedee]

MSI Z97A Gaming 7 desktop motherboard

I already have a hook that signs my kernels after installation. To be clear, you're saying that the hook you've pasted signs them at an intermediary step before the final signing? Is this just a workaround for now or is this a permanent solution that should be added to the wiki?

[patrakov]

Yes, I have two signing hooks. One signs the vmlinuz files and runs before mkinitcpio, and another signs the files produced by mkinitcpio.

Regarding the "workaround or official solution" question, can't answer - https://github.com/systemd/systemd/issues/25336#issuecomment-1312479870 implies that the kernels should be signed before being added to the UKI (which should then be signed again), but until my "is it safe, from the perspective of an evil maid being able to deploy an untrusted initrd" question is answered, I cannot recommend this.

EDIT: the exact evil-maid scenario is: take the disk, copy the valid UKI from the EFI partition, extract the kernel (which now has a valid signature!), command line, and initrd from there, create a modified initrd, create a new boot loader entry that references them, instead of the original one, put the disk back, let me boot the trojaned initrd.

[domker]

I have the same problem after upgrading to systemd 252.1-1. The system won't boot and it says "Access Denied". The only thing that helps is a physical reset with the button and disabling secure boot in UEFI.

Operating System: Arch Linux KDE Plasma Version: 5.26.3 KDE Frameworks Version: 5.99.0 Qt Version: 5.15.7 Kernel Version: 6.0.8-arch1-1 (64-bit) Graphics Platform: X11 Processors: 8 × Intel® Core™ i7-4790K CPU @ 4.00GHz Memory: 15.6 GiB of RAM Graphics Processor: NVIDIA GeForce GTX 980/PCIe/SSE2 Manufacturer: ASUS Firmware: UEFI 2.31 (American Megatrends 4.655)

On the second computer also with an newer Asus motherboard and newer firmware (UEFI 2.50 American Megatrends 5.12) after updating to systemd 251.1-1, the system starts, but there is an error "Failed to reconnect handle 352. ignoring : Security Policy Violation".

In both cases, I use the "sbupdate-git" script to sign the image. So far everything worked fine.

/etc/sbupdate.conf:

BACKUP=0
EXTRA_SIGN=('/boot/EFI/BOOT/BOOTX64.EFI' '/boot/EFI/systemd/systemd-bootx64.efi')
CMDLINE_DEFAULT="root=PARTUUID=my_part_uuid rw mitigations=off tsx=on lsm=lockdown,yama,apparmor,bpf quiet"
CONFIGS["linux"]="linux linux-fallback"

I'm generally not familiar with Secure Boot to somehow fix the problem myself, so after upgrading to systemd 252.1-1 I disabled Secure Boot completely, until the problem was solved with another systemd update. I'm sure many other users will do the same. Attempting to increase security in systemd regarding Secure Boot will result in even worse results, i.e. disabling protection altogether.

As the example of two of my computers shows, depending on the firmware version, either the system cannot be started completely, or UEFI itself adds an security exception and conditionally allows booting, but with an error message.

[inthreedee]

I have tried testing the posted workaround but I still receive the error message. I've signed my vmlinuz files in /usr/lib/modules, reinstalled the kernels and verified that the resulting UKIs are also signed, but no dice. I'm not sure what I've done wrong as it sounds like this is supposed to solve the problem?

@patrakov have I missed a step?

[medhefgo]

I am pretty sure that secure boot gives you no protection against evil catmaids unless you also employ encryption+tpm. Naked secure boot is only really an anti-malware protection.

Please give #25409 a try.

[patrakov]

It boots (with the kernel-signing hook removed), here is the debug output:

[ckujau]

Similar here when trying to boot a converted Fedora 39 via Libvirt, running on a Fedora 39 host:

$ virsh version
Compiled against library: libvirt 9.7.0
Using library: libvirt 9.7.0
Using API: QEMU 9.7.0
Running hypervisor: QEMU 8.1.3

$ virsh start --console fedora0
BdsDxe: loading Boot0003 "Linux Boot Manager" from HD(2,GPT,44A7D773-FAE9-4924-823F-2F59058CF5CA,0x1800,0x32000)/\EFI\systemd\systemd-bootx64.efi
BdsDxe: failed to load Boot0003 "Linux Boot Manager" from HD(2,GPT,44A7D773-FAE9-4924-823F-2F59058CF5CA,0x1800,0x32000)/\EFI\systemd\systemd-bootx64.efi: Access Denied
BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x3)/Pci(0x0,0x0)
BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x3)/Pci(0x0,0x0): Access Denied
BdsDxe: No bootable option or device was found.
BdsDxe: Press any key to enter the Boot Manager Menu.

 Standard PC (Q35 + ICH9, 2009) 
 pc-q35-8.1                                          2.00 GHz
 edk2-20231122-14.fc39                               2048 MB RAM

Disabling Secure Boot in the Libvirt firmware helps and the system boots via systemd-boot just fine:

$ bootctl
System:
      Firmware: UEFI 2.70 (EDK II 1.00)
 Firmware Arch: x64
   Secure Boot: disabled
  TPM2 Support: no
  Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 254.7-1.fc39
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Load drop-in drivers
               ✓ Support Type #1 sort-key field
               ✓ Support @saved pseudo-entry
               ✓ Support Type #1 devicetree field
               ✓ Enroll SecureBoot keys
               ✓ Retain SHIM protocols
               ✓ Boot loader sets ESP information
          ESP: /dev/disk/by-partuuid/44a7d773-fae9-4924-823f-2f59058cf5ca
         File: └─/EFI/systemd/systemd-bootx64.efi
[...]