search

fedora-silverblue / issue-tracker

Fedora Silverblue issue tracker
https://fedoraproject.org/atomic-desktops/silverblue/
125 stars 3 forks source link

Unable to boot rawhide after latest deployment, missing bli.mod file #587

Closed haecker-felix closed 1 week ago

haecker-felix commented 1 month ago

Describe the bug System won't boot anymore, grub displays this for few milliseconds before the system reboots into uefi setup. 

error: ../../grub-core/fs/fshelp.c:257:file 'grub2/x86_64-efi/bli.mod' not found.

To Reproduce Please describe the steps needed to reproduce the bug:

  1. Upgrade from 40 to rawhide
  2. Add some new overlay packages
  3. Reboot to apply them

Expected behavior The system boots

OS version: Since I'm not able anymore to boot the system, no idea, but IIRC it was 20240810

Additional context Related: https://discussion.fedoraproject.org/t/can-t-boot-after-update/128493/6

samcday commented 1 month ago

A couple of folks have been reporting this for a few days now in the Silverblue Matrix channel as well.

haecker-felix commented 1 month ago

I managed to boot my system again with some tricks:

  1. Flash a live system to a usb drive and boot from it
  2. Mount the ANACONDA EFI partition of the usb drive
  3. Edit the ANACONDA/EFI/BOOT/grub.cfg file
  4. Now insert the ostree menu entries from your silverblue system. You can find those in /boot/grub2/grub.cfg
  5. Change linux16 to linux and initrd16 to initrd

After all my grub.cfg of the usb drive looks like this:

set default="1"

function load_video {
  insmod efi_gop
  insmod efi_uga
  insmod video_bochs
  insmod video_cirrus
  insmod all_video
}

load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2

set timeout=60
### END /etc/grub.d/00_header ###

insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root bb2467e2-39a4-4325-89b5-2b060005b4ba
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=boot F225-FB4F

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Start Fedora-Workstation-Live 40' --class fedora --class gnu-linux --class gnu --class os {
    linuxefi /images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-40-1-14  rd.live.image quiet rhgb
    initrdefi /images/pxeboot/initrd.img
}
menuentry 'Test this media & start Fedora-Workstation-Live 40' --class fedora --class gnu-linux --class gnu --class os {
    linuxefi /images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-40-1-14  rd.live.image rd.live.check quiet
    initrdefi /images/pxeboot/initrd.img
}
submenu 'Troubleshooting -->' {
    menuentry 'Start Fedora-Workstation-Live 40 in basic graphics mode' --class fedora --class gnu-linux --class gnu --class os {
        linuxefi /images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-40-1-14  rd.live.image nomodeset quiet rhgb
        initrdefi /images/pxeboot/initrd.img
    }
}

### BEGIN /etc/grub.d/15_ostree ###
menuentry 'Fedora Linux Rawhide.20240811.n.0 (Silverblue Prerelease) (ostree:0)' --class gnu-linux --class gnu --class os --unrestricted 'ostree-0-bb2467e2-39a4-4325-89b5-2b060005b4ba' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root bb2467e2-39a4-4325-89b5-2b060005b4ba
linux /ostree/fedora-8f02519688a81b667d2e75f77374a2f3924609b8397ab155cfdbdf4b5a30366f/vmlinuz-6.11.0-0.rc2.20240809gitee9a43b7cfe2.27.fc41.x86_64 rhgb quiet root=UUID=74c6b768-9643-42fa-999c-ffd3eebeba3b rootflags=subvol=root rw ostree=/ostree/boot.0/fedora/8f02519688a81b667d2e75f77374a2f3924609b8397ab155cfdbdf4b5a30366f/0
initrd /ostree/fedora-8f02519688a81b667d2e75f77374a2f3924609b8397ab155cfdbdf4b5a30366f/initramfs-6.11.0-0.rc2.20240809gitee9a43b7cfe2.27.fc41.x86_64.img
}
menuentry 'Fedora Linux Rawhide.20240811.n.0 (Silverblue Prerelease) (ostree:1)' --class gnu-linux --class gnu --class os --unrestricted 'ostree-1-bb2467e2-39a4-4325-89b5-2b060005b4ba' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root bb2467e2-39a4-4325-89b5-2b060005b4ba
linux /ostree/fedora-8f02519688a81b667d2e75f77374a2f3924609b8397ab155cfdbdf4b5a30366f/vmlinuz-6.11.0-0.rc2.20240809gitee9a43b7cfe2.27.fc41.x86_64 rhgb quiet root=UUID=74c6b768-9643-42fa-999c-ffd3eebeba3b rootflags=subvol=root rw ostree=/ostree/boot.0/fedora/8f02519688a81b667d2e75f77374a2f3924609b8397ab155cfdbdf4b5a30366f/1
initrd /ostree/fedora-8f02519688a81b667d2e75f77374a2f3924609b8397ab155cfdbdf4b5a30366f/initramfs-6.11.0-0.rc2.20240809gitee9a43b7cfe2.27.fc41.x86_64.img
}
### END /etc/grub.d/15_ostree ###

Now reboot your system, boot from the usb drive, and choose the ostree menu entry. Your system should boot with this.

haecker-felix commented 1 month ago 
haeckerfelix@fedora:~$ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/rawhide/x86_64/silverblue
                  Version: Rawhide.20240811.n.0 (2024-08-11T06:10:25Z)
               BaseCommit: 57afc984a2fb21d58bcab52a7e7f1e1d39cb8c5002b222268403a7440fcf01ee
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
      RemovedBasePackages: firefox firefox-langpacks 128.0-3.fc41 gnome-terminal-nautilus gnome-terminal 3.50.1-10.fc41
          LayeredPackages: gnome-console syncthing sysprof

  fedora:fedora/rawhide/x86_64/silverblue
                  Version: Rawhide.20240811.n.0 (2024-08-11T06:10:25Z)
                   Commit: 57afc984a2fb21d58bcab52a7e7f1e1d39cb8c5002b222268403a7440fcf01ee
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1

Now if somebody can tell me how I can recover my grub somehow from a booted system I'd be really grateful :)

haecker-felix commented 1 month ago 
sudo grub2-install /dev/nvme0n1

fails because it can't find /usr/lib/grub/x86_64-efi/modinfo.sh - for which I'm not sure

samcday commented 1 month ago

Interesting. My working F40 Silverblue install does not have a /usr/lib/grub/x86_64-efi/ directory, but rather /usr/lib/grub/i686-pc/.

Kinda seems like a switch from BIOS compatibility mode to x64-uefi has happened, maybe that's part of the root cause for this issue?

samcday commented 1 month ago

how I'm supposed to install the missing package, given I'm not able to add new overlays and reboot because I have no working bootloader

sudo rpm-ostree usroverlay lets you unlock your current deployment to resolve any package state issues in the live install, at which point you cshould then be able to sudo grub2-install your way to victory

haecker-felix commented 1 month ago

I managed to work around it by installing the missing package in a toolbox and copy the the x86_64-efi folder to the host system, and try to reinstall grub with

sudo grub2-install /dev/nvme0n1 --directory ./x86_64-efi --force

which unfortunately results in the same missing bli.mod issue.

kanru commented 1 month ago

I was able to reproduce this in a VM.

Content in /boot/efi sha256sum $(find efi -type f|sort):

5ec41801add3ff171fedc68bb343b794c330756796e12af23d28ff9fc5cadb79  efi/EFI/BOOT/BOOTIA32.EFI
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  efi/EFI/BOOT/BOOTX64.EFI
8411f49e3299eafb92eb44992e373eef6be17943505df7431c901c95cdda3e1f  efi/EFI/BOOT/fbia32.efi
e6ac4f6428bfd524bba6ccf69bef22538c1114e9ae3be23e084a4304df51064b  efi/EFI/BOOT/fbx64.efi
43f8aba93558f40b14e10d2cace57111014b041b40e3f57344e40681cfde4fd5  efi/EFI/fedora/BOOTIA32.CSV
282a3b2872bebcf3d1ba42fe17c17ebe32a60759a20f058b5e960e542785fe82  efi/EFI/fedora/BOOTX64.CSV
7d1e0b1f7f00a1c1b2036dddd103f4cd44dc84f2e997feff86b583a377f0ac90  efi/EFI/fedora/grub.cfg
80ffcfaedc719239e9e7cf28cc1454abca3462dd0c429fe2b2eac387531a9d80  efi/EFI/fedora/grubia32.efi
edc235892fe2fc0f9de620f7af77bda1db288a221af2ec858fd9736e5285cdfa  efi/EFI/fedora/grubx64.efi
c0aca1c783936992460e2857749cc647894a08c61fd17992f510d4be6b35e050  efi/EFI/fedora/mmia32.efi
83a49149f5a7060d0de43de3a0c683d99f3c58c418ab651ac9c254b4efaef0b8  efi/EFI/fedora/mmx64.efi
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  efi/EFI/fedora/shim.efi
5ec41801add3ff171fedc68bb343b794c330756796e12af23d28ff9fc5cadb79  efi/EFI/fedora/shimia32.efi
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  efi/EFI/fedora/shimx64.efi

Content in first deployment be665cc30fd18ffeb0cf624ceee92441e983c70cd559e4913dbb2769335225b9.0:

cd /mnt/root/root/ostree/deploy/fedora/deploy/be665cc30fd18ffeb0cf624ceee92441e983c70cd559e4913dbb2769335225b9.0/usr/lib/bootupd/updates/
sha256sum $(find EFI -type f|sort)
5ec41801add3ff171fedc68bb343b794c330756796e12af23d28ff9fc5cadb79  EFI/BOOT/BOOTIA32.EFI
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  EFI/BOOT/BOOTX64.EFI
8411f49e3299eafb92eb44992e373eef6be17943505df7431c901c95cdda3e1f  EFI/BOOT/fbia32.efi
e6ac4f6428bfd524bba6ccf69bef22538c1114e9ae3be23e084a4304df51064b  EFI/BOOT/fbx64.efi
43f8aba93558f40b14e10d2cace57111014b041b40e3f57344e40681cfde4fd5  EFI/fedora/BOOTIA32.CSV
282a3b2872bebcf3d1ba42fe17c17ebe32a60759a20f058b5e960e542785fe82  EFI/fedora/BOOTX64.CSV
12d079fbcc2c4ede3564d1fc7fbd802fd7d369ff2554306348b367867c3c2a5b  EFI/fedora/grubia32.efi
9d6ab71f344764bcab32bab8236a15ae7d006f49884681f1bdcc628d8aaee100  EFI/fedora/grubx64.efi
c0aca1c783936992460e2857749cc647894a08c61fd17992f510d4be6b35e050  EFI/fedora/mmia32.efi
83a49149f5a7060d0de43de3a0c683d99f3c58c418ab651ac9c254b4efaef0b8  EFI/fedora/mmx64.efi
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  EFI/fedora/shim.efi
5ec41801add3ff171fedc68bb343b794c330756796e12af23d28ff9fc5cadb79  EFI/fedora/shimia32.efi
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  EFI/fedora/shimx64.efi

Content in second deployment 8d93cfe1ee0627cec03adc302b249c821950acb9565bc936a1296b403d787ac3.0:

cd /mnt/root/root/ostree/deploy/fedora/deploy/8d93cfe1ee0627cec03adc302b249c821950acb9565bc936a1296b403d787ac3.0/usr/lib/bootupd/updates
sha256sum $(find EFI -type f|sort)
5ec41801add3ff171fedc68bb343b794c330756796e12af23d28ff9fc5cadb79  EFI/BOOT/BOOTIA32.EFI
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  EFI/BOOT/BOOTX64.EFI
8411f49e3299eafb92eb44992e373eef6be17943505df7431c901c95cdda3e1f  EFI/BOOT/fbia32.efi
e6ac4f6428bfd524bba6ccf69bef22538c1114e9ae3be23e084a4304df51064b  EFI/BOOT/fbx64.efi
43f8aba93558f40b14e10d2cace57111014b041b40e3f57344e40681cfde4fd5  EFI/fedora/BOOTIA32.CSV
282a3b2872bebcf3d1ba42fe17c17ebe32a60759a20f058b5e960e542785fe82  EFI/fedora/BOOTX64.CSV
12d079fbcc2c4ede3564d1fc7fbd802fd7d369ff2554306348b367867c3c2a5b  EFI/fedora/grubia32.efi
9d6ab71f344764bcab32bab8236a15ae7d006f49884681f1bdcc628d8aaee100  EFI/fedora/grubx64.efi
c0aca1c783936992460e2857749cc647894a08c61fd17992f510d4be6b35e050  EFI/fedora/mmia32.efi
83a49149f5a7060d0de43de3a0c683d99f3c58c418ab651ac9c254b4efaef0b8  EFI/fedora/mmx64.efi
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  EFI/fedora/shim.efi
5ec41801add3ff171fedc68bb343b794c330756796e12af23d28ff9fc5cadb79  EFI/fedora/shimia32.efi
4773d74d87c2371a25883b59a3b6d98d157de46933676706d215015b1130f2d1  EFI/fedora/shimx64.efi

These 2 files do not match any files in either deployments:

80ffcfaedc719239e9e7cf28cc1454abca3462dd0c429fe2b2eac387531a9d80  efi/EFI/fedora/grubia32.efi
edc235892fe2fc0f9de620f7af77bda1db288a221af2ec858fd9736e5285cdfa  efi/EFI/fedora/grubx64.efi
kanru commented 1 month ago

It boots again if I replace the two files with the ones from either deployments.

This checksum matches the file in grub2-efi-x64-2.12-4.fc41.x86_64.rpm

9d6ab71f344764bcab32bab8236a15ae7d006f49884681f1bdcc628d8aaee100  EFI/fedora/grubx64.efi

So where did this one come from?

edc235892fe2fc0f9de620f7af77bda1db288a221af2ec858fd9736e5285cdfa  efi/EFI/fedora/grubx64.efi
haecker-felix commented 1 month ago

Good catch @kanru, can confirm that swapping out those two files makes the system bootable again :+1:

0unknwn commented 4 weeks ago

Thanks, @kanru! This resolved the issue for me as well. I hope this only affects Rawhide. I find these bootloader issues quite concerning, because they kind of defeat the purpose of using Silverblue in the first place.

(I deployed Silverblue to family members to avoid the situation of getting send an unbootable device, but maybe it is still too experimental and should be marked as such.)

samcday commented 4 weeks ago

Thanks, @kanru! This resolved the issue for me as well. I hope this only affects Rawhide. I find these bootloader issues quite concerning, because they kind of defeat the purpose of using Silverblue in the first place.

(I deployed Silverblue to family members to avoid the situation of getting send an unbootable device, but maybe it is still too experimental and should be marked as such.)

Rawhide is indeed too experimental for the use case you're describing. It has indeed been marked as such since always, in very strongly worded warnings everywhere, and in fact implied by the name. Not sure what you're expecting here? :)

0unknwn commented 4 weeks ago

If it’s only happening on Rawhide it is fine. But grub was broken on normal installations as well, so I guess the boot process is really a weak point, but hopefully this is resolved with bootupd.

My thought was that it would be relatively save to rebase to Rawhide, because I could always just rollback to a working deployment, but this was maybe overestimating the power of ostree base distros. My family members a running the stable version of course.

samcday commented 4 weeks ago

If it’s only happening on Rawhide it is fine.

I'm unaware of any of the reports in this issue, the linked discussion or on Matrix indicating that this is affecting F40 stable releases. Are you seeing something different?

But grub was broken on normal installations as well

What are you referring to here?

My thought was that it would be relatively save to rebase to Rawhide, because I could always just rollback to a working deployment, but this was maybe overestimating the power of ostree base distros.

Yes, I think I've run into this mentality a few times now in the Matrix channel as well. I think perhaps the documentation and general stance could (should) be more strongly voiced that Rawhide is not suitable for daily use in any kind of environment that is relied upon. I consider myself a highly chaotic #YOLO hacker man that constantly tests/pushes the limits of my software environment, and I've never ended up with a bricked install, because I don't use Rawhide ;)

My family members a running the stable version of course.

:+1: I'd strongly recommend you do the same. If you were using Rawhide to access upgraded packages, consider selectively installing the things you care about from Rawhide using the fedora-repos-rawhide package:

sudo rpm-ostree install -A dnf fedora-repos-rawhide
sudo rpm-ostree usroverlay
sudo dnf upgrade -y --enablerepo=rawhide <your package>
0unknwn commented 4 weeks ago

Are you seeing something different?

No, and that is reassuring.

What are you referring to here?

The outdated grub thing, where you needed to copy a file to make it work.

Yes, I think I've run into this mentality a few times now in the Matrix channel as well. I think perhaps the documentation and general stance could (should) be more strongly voiced that Rawhide is not suitable for daily use in any kind of environment that is relied upon.

Seems to be a common misconception. Maybe rpm-ostree could even warn when rebasing to rawhide.

sudo rpm-ostree install -A dnf fedora-repos-rawhide
sudo rpm-ostree usroverlay
sudo dnf upgrade -y --enablerepo=rawhide <your package>

What a nice way to test things. Wasn’t aware of this possibility. Thanks!

samcday commented 4 weeks ago

The outdated grub thing, where you needed to copy a file to make it work.

Ah yes, I thought you might be referring to this. I certainly agree with your earlier sentiment about the fragility of bootloaders. There's definitely been a string of unfortunate coincidences this year in that regard, too ;) FWIW, that's an issue with shim, and not GRUB itself. So I would categorize that as a different kind of failure (Secure Boot compatibility, not GRUB) . I also hope that bootupd proves to be a nice coherent place to bullet-proof this stuff.

Seems to be a common misconception. Maybe rpm-ostree could even warn when rebasing to rawhide.

Yeah, I like that idea :)

What a nice way to test things. Wasn’t aware of this possibility. Thanks!

You can also make it more permanent if you roll your own image: https://github.com/samcday/workstation-config/blob/41dd33e/Dockerfile#L118-L119

garrett commented 4 weeks ago

I can confirm that the fix to boot again is to basically do this:

  1. Boot into a Live USB version of Workstation
  2. copy the grubx64.efi file (see below how to get it) with the sha256sum 9d6ab71f344764bcab32bab8236a15ae7d006f49884681f1bdcc628d8aaee100 to your mounted efi partition (from your computer's internal storage, not the live USB one) as EFI/fedora/grubx64.efi
  3. Reboot back into your normal installtion
  4. Rebase back to non-rawhide rpm-ostree rebase fedora:fedora/40/x86_64/silverblue

...no messing with grub configuration files needed.

As to get the correct grubx64.efi file? I used toolbox like this:

  1. toolbox create -i fedora-toolbox:rawhide
  2. toolbox enter fedora-toolbox-rawhide
  3. sudo dnf install grub2-efi-x64 -y
  4. cp /boot/efi/EFI/fedora/grubx64.efi .
  5. exit

(You can also find the RPM, download it, use rpm2cpio and such, but I figured the above is easier than hunting down the right file.)

Now you should have the correct grubx64.efi, ready to copy to the correct place on your internal storage (the mounted EFI partition, under the EFI/fedora/ subdirectories). Follow the steps above (first set) to fix your system.

freebritney132 commented 4 weeks ago

grubx64.efi.txt Here's a copy of grubx64.efi with the sha256 hash 9d6ab71f344764bcab32bab8236a15ae7d006f49884681f1bdcc628d8aaee100 if anyone needs it, just remove the .txt extension since github doesn't allow you to upload .efi files.

kanru commented 4 weeks ago

For anyone that needs to perform the recovery process, could you backup and share the sha256sum of the not working grubx64.efi file? I'd like to see if they all have the same checksum or all have different ones.

Also, does anyone have any idea what rpm-ostree process can affect grub?

yodatak commented 3 weeks ago

@kanru Mine buggy efi 3.5 MB instead of 4.1 MB 77b95c266b2cf39b10c1cdbf3f828ec9ddfbed3536fa6fda90debae16f60c247 grubx64.efi stat grubx64.efi File: grubx64.efi Size: 3530048 Blocks: 6896 IO Block: 4096 regular file Device: 259,1 Inode: 323 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/liveuser) Gid: ( 1000/liveuser) Context: system_u:object_r:dosfs_t:s0 Access: 2024-08-15 00:00:00.000000000 -0400 Modify: 1979-12-31 23:00:00.000000000 -0500 Change: 1979-12-31 23:00:00.000000000 -0500 Birth: 2023-09-24 18:31:26.700000000 -0400

grubx64.efi.buggy.txt

TheGreatDeadOne commented 3 weeks ago

Does this bug affect the new branch 41?

kanru commented 3 weeks ago

Yes, it's reproducible with the latest branched F41.

It looks like it's not a corrupted grub, but a incompatible one.

After refresh F40 install, the grubx64.efi has the "bad" checksum. It's from grub2-2.06-119.fc40 (grubx64.efi info)

root@fedora:/var/home/kanru# sha256sum /boot/efi/EFI/fedora/grubx64.efi 
edc235892fe2fc0f9de620f7af77bda1db288a221af2ec858fd9736e5285cdfa  /boot/efi/EFI/fedora/grubx64.efi

root@fedora:/var/home/kanru# stat /boot/efi/EFI/fedora/grubx64.efi 
  File: /boot/efi/EFI/fedora/grubx64.efi
  Size: 3968320     Blocks: 7752       IO Block: 4096   regular file
Device: 252,1   Inode: 143         Links: 1
Access: (0700/-rwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:dosfs_t:s0
Access: 2024-08-16 09:00:00.000000000 +0900
Modify: 1980-01-01 09:00:00.000000000 +0900
Change: 1980-01-01 09:00:00.000000000 +0900
 Birth: 2024-08-16 10:05:04.020000000 +0900

After rebase to F41 and reboot - same content - bootable

After rebase to F41 and layering syncthing - same content - not bootable

latenightdef commented 3 weeks ago

I upgraded to Kinoite 41 and it was broken too, however running toolbox create -i fedora-toolbox:rawhide from Fedora KDE 40 live usb errored.

With toolbox already manually installed in the live session, I found another alternative solution for acquiring the grubx64.efi file:

  1. podman run -it -v /home/liveuser:/home/liveuser fedora-toolbox:rawhide
  2. Disable SELinux in the live session by running sudo setenforce 0 from another terminal window
  3. dnf install grub2-efi-x64 -y No need for sudo since we're already root in the container
  4. cp /boot/efi/EFI/fedora/grubx64.efi .
  5. exit

After copying the correct grubx64.efi to my system's EFI System Partition, in <mounted ESP>/EFI/fedora, my F41 Kinoite finally booted again, now I can either 1) keep using 41 and try this fix again if bootloader breaks 2) rebase back to 40

samcday commented 3 weeks ago

keep using 41 and try this fix again if bootloader breaks

Well, to continue my slightly off-topic rant from earlier, right now the F41 branch, for all intents and purposes, IS rawhide. It was branched only a couple of days ago, and is (evidently) not yet ready for release and general consumption.

So I think you, and likely 99% of the folks in this thread so far, should be rebasing back to F40 and waiting for a proper F41 release :)

F41 is still a week and a bit from beta freeze, and not targeted for release until mid October.

kanru commented 3 weeks ago

It looks like it's not a corrupted grub, but a incompatible one.

I identified the issue. GRUB 2.12 introduced two incompatible changes in the generated grub.cfg

First, the bli module is not available in previous GRUB

### BEGIN /etc/grub.d/25_bli ###
if [ "$grub_platform" = "efi" ]; then
  insmod bli
fi
### END /etc/grub.d/25_bli ###

Second, fwsetup gained a new --is-supported parameter

### BEGIN /etc/grub.d/30_uefi-firmware ###
if [ "$grub_platform" = "efi" ]; then
    fwsetup --is-supported
    if [ "$?" = 0 ]; then
        menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' {
            fwsetup
        }
    fi
fi
### END /etc/grub.d/30_uefi-firmware ###

Both can be fixed by wrapping the command in a test expression. I opened https://bugzilla.redhat.com/show_bug.cgi?id=2305291

cgwalters commented 3 weeks ago

I think you'll only hit this with systems that are still using the dynamic grub config generation; with bootupd we now heavily emphasize static grub configs.

But yes, this is also another case on the flip side where rpm-ostree/bootc not updating the bootloader by default can bite folks, xref https://github.com/fedora-silverblue/issue-tracker/issues/120

Yonnji commented 3 weeks ago

I had this too. I did rebase to F41, rebooted without a problem, installed few more packages and got boot broken. It also very difficult to see an error message, because it stays only for a 100ms and then instantly reboots into UEFI settings. I had to record slow motion video of the error message with a phone, and then go to the video editor on the phone in order to finally see the actual message.

MindlessDreams commented 3 weeks ago

2. to your mounted efi partition (from your computer's internal storage, not the live USB one) as EFI/fedora/grubx64.efi

Could you please specify how to do this? I rebased to 41 to checkout the accent colors, all worked fine, today rebooted and can't restore the system. I got a Fedora 40 live usb running once in it I open terminal, sudo myself, cd to /boot/efi/EFI/fedora/, delete grubx64.efi, then copy in the same directory the "correct one", reboot and nothing - same error. What do I do wrong, how do I do it correctly? Thank you!

latenightdef commented 3 weeks ago
  1. to your mounted efi partition (from your computer's internal storage, not the live USB one) as EFI/fedora/grubx64.efi

Could you please specify how to do this? I rebased to 41 to checkout the accent colors, all worked fine, today rebooted and can't restore the system. I got a Fedora 40 live usb running once in it I open terminal, sudo myself, cd to /boot/efi/EFI/fedora/, delete grubx64.efi, then copy in the same directory the "correct one", reboot and nothing - same error. What do I do wrong, how do I do it correctly? Thank you!

That looks like the Live USB's EFI system partition, try finding your actual system's EFI system partition and mount it at /mnt by:

  1. Run lsblk -f in the Live USB's terminal. You should see something that look similar to this, minus the mountpoints since I'm running this on a working system and not the Live USB. Notice the nvme0n1p1 being FAT32, that means the /dev/nvme0n1p1 is the actual system's EFI system partition. 
    
nvme0n1                                                                                       
├─nvme0n1p1 vfat   FAT32                  xxxx-xxxx                            1009.4M     1% /boot/efi
├─nvme0n1p2 ext4   1.0                    xxxxxxxx-xxxx-xxxx-xxxx-e355f3c7c050  751.3M    16% /boot
├─nvme0n1p3 btrfs                         xxxxxxxx-xxxx-xxxx-xxxx-1bbc9f4aa57f     26G    45% /var
│                                                                                             /sysroot/ostree/deploy/fedora/var
│                                                                                             /usr
│                                                                                             /etc
│                                                                                             /
│                                                                                             /sysroot
├─nvme0n1p4 btrfs                         xxxxxxxx-xxxx-xxxx-xxxx-2d5d0c148f94   17.2G     9% /var/home
├─nvme0n1p5 ext4   1.0   My Flatpak    xxxxxxxx-xxxx-xxxx-xxxx-f1acb4938708      7G    59% /var/home/me/.var
└─nvme0n1p6 ext4   1.0   My Data      xxxxxxxx-xxxx-xxxx-xxxx-23601be3e058   27.4G    67% /run/media/me/Data

2. Mount /dev/nvme0n1p1 at /mnt by running `sudo mount /dev/nvme0n1p1 /mnt`. Make sure to replace `/dev/nvme0n1p1` accordingly!
3. Follow [my guide above](https://github.com/fedora-silverblue/issue-tracker/issues/587#issuecomment-2292675657) and put the correct grubx64.efi in `/mnt/EFI/fedora`

Edit: added detailed howto
MindlessDreams commented 3 weeks ago

Edit: added detailed howto

THANK YOU!!! Back in and rebased to fedora 40 :)))

travier commented 3 weeks ago

Thanks @kanru for the investigation here.

One option that we have pending bootupd being enabled by default is to patch out those configs for F41 until https://bugzilla.redhat.com/show_bug.cgi?id=2305291 is fixed.

travier commented 2 weeks ago

Untested, tentative workaround for this issue:

travier commented 1 week ago

I've merged the PRs above and they are now available in Rawhide and F41 and this "fixed" (i.e. worked around the issue) as far as I could test. We'll revert those workarounds once we get bootupd doing bootloader updates by default.