Use bootupd and remove ostree-grub2 (was: All deployments are shown twice in grub)

[YaLTeR]

F33, ThinkPad T495s

[felipeborges]

I can reproduce this too.

[jlebon]

See https://fedoraproject.org/wiki/Common_F31_bugs#On_Fedora_Silverblue.2FIoT.2C_the_GRUB_menu_shows_duplicate_entries (I've added an "Update:" section there).

[YaLTeR]

See https://fedoraproject.org/wiki/Common_F31_bugs#On_Fedora_Silverblue.2FIoT.2C_the_GRUB_menu_shows_duplicate_entries (I've added an "Update:" section there).

So... should I migrate to BLS? If I should then maybe it should rather be done by default in Silverblue?

[jlebon]

So... should I migrate to BLS? If I should then maybe it should rather be done by default in Silverblue?

If you can, yes. I'm not sure if it's worth forcing a migration at this point because of the potential risks (see e.g. https://discussion.fedoraproject.org/t/boot-entries-gone-after-upgrade/8026). And as mentioned in the wiki, the migration script only works on EFI because it might not be safe to do it automatically on BIOS (see https://github.com/ostreedev/ostree/pull/2044#issuecomment-608316436).

[Jmennius]

What about systems that did not upgrade to F31 (clean install F31+)? I did not find any place that creates /boot/grub2/.grub2-blscfg-supported for new installations. I have BLS enabled (Silverblue) but there is no such file (probably this system started with F31).

I am also concerned that .grub2-switch-to-blscfg does not work well with https://fedoraproject.org/wiki/Changes/UnifyGrubConfig.

cc @jlebon @martinezjavier

[dustymabe]

I'm seeing this on a fresh install of F35 Silverblue. I guess we need to track down what used to create the /boot/grub2/.grub2-blscfg-supported and see why it's not getting created any longer.

[dustymabe]

Found it: https://src.fedoraproject.org/rpms/grub2/blob/9fdaa794e06846f1426bd15360401a8b1a5017a9/f/0065-Add-grub2-switch-to-blscfg.patch#_362

[jlebon]

I'm seeing this on a fresh install of F35 Silverblue. I guess we need to track down what used to create the /boot/grub2/.grub2-blscfg-supported and see why it's not getting created any longer.

I think... nothing ever actually created that for new installs. The idea we discussed was to have Anaconda do it, but it looks like we never actually followed through with it.

The other option is to just assume GRUB is new enough, but that can break people upgrading from an old install with old GRUB. The failure mode here is particularly bad: they'll get no boot entries.

Maybe we just need to be really loud about it and say that you need to make sure that your GRUB is up to date before $date using either the migration script or bootupd, and then we rip out this check.

[jlebon]

Maybe we just need to be really loud about it and say that you need to make sure that your GRUB is up to date before $date using either the migration script or bootupd, and then we rip out this check.

A prerequisite for that is of course to ship bootupd in FSB, which we currently don't.

[jlebon]

Yeah, I think bootupd is probably the cleanest path forward for this and for possible future issues like it. So my strawman is something like:

• add bootupd in FSB
• teach Anaconda to call out to bootupctl
• teach 15_ostree to read bootupd metadata to know if GRUB supports blscfg (but really, if bootupd metadata exists, it surely does, so this might just be test -f /boot/.bootupd-state.json)
• tell upgrading users that this bug is fixed in the latest bootloader, and they can update using bootupctl adopt-and-update

@cgwalters @martinezjavier WDYT?

[travier]

This might need a Fedora Change for F36 (Silverblue & Kinoite).

[martinezjavier]

@cgwalters @martinezjavier WDYT?

This makes sense to me. In fact, it would be great if even the non-ostree Fedora variants could adopt bootupd.

[travier]

The change deadline is mid January if we think this is a self contained change: https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html

[ormandj]

This would be quite a nice change - I was surprised to see the duplicate entries on a clean 35 Silverblue installation, and ran across this issue while researching resolution.

[travier]

How do all of you feel about trying to make that happen for F37 and maybe opt-in for F36 (ship bootupd but not enabled by default)? I can start the change request.

[tpopela]

It was great to see @martinezjavier 's option and if he's in favor, then I'm as well. Thank you @travier for looking into this!

[jlebon]

How do all of you feel about trying to make that happen for F37 and maybe opt-in for F36 (ship bootupd but not enabled by default)? I can start the change request.

SGTM, thanks for picking this up!

[tpopela]

It would be good to tackle this after the recent round of the grub2 CVEs. We might also need to fix https://github.com/coreos/bootupd/issues/108 . But it looks like there was some work done by @cgwalters , but might be not finished - what's the current state of it Colin?

[koko-ng]

I ran into this problem while debugging why firmware updates weren't applied. If anyone runs into the same issue, before there is a cleaner fix it's possible to update shim like this:

sudo rpm-ostree usroverlay 
wget https://kojipkgs.fedoraproject.org//packages/shim/15.6/1/x86_64/shim-x64-15.6-1.x86_64.rpm
sudo rpm -i  --reinstall shim-x64-15.6-1.x86_64.rpm

[dustymabe]

I'm not sure if rpm-ostree usroverlay is what you want:

       usroverlay
           Mount a writable overlay filesystem on /usr which is
           active only for the remainder of the system boot. This
           is intended for development, testing, and debugging.
           Changes will not persist across upgrades, or rebooting
           in general.

[koko-ng]

It's just so that rpm can write to the database, of course it will be discarded on next reboot, but the changes to /bin/efi will effectively be kept as they are not managed by ostree and as rpm reinstall the same version of the package, the content of the persistent database (the one managed by rpm-ostree) is now coherent with the content of /bin/efi.

[travier]

With https://pagure.io/workstation-ostree-config/pull-request/288, we now have bootupd in Rawhide but not enabled by default.

Let's track what's missing here and make a Change request for F38.

For reference, the commit message from Jonathan:

We need to make it easier to update the bootloader on these variants
because unlike on traditional systems, it's not updated automatically
with the rest of the system. Add bootupd for that.

This would allow fixing issues like:
- https://github.com/coreos/rpm-ostree/issues/3715
- https://github.com/fedora-silverblue/issue-tracker/issues/120#issuecomment-979268679

It won't be enabled by default and as mentioned in that comment requires
work in Anaconda to be seamless. But at least with this users should be
able to adopt and update:

https://github.com/coreos/bootupd/blob/main/README-design.md

See also the tracker issue where we did this for Fedora CoreOS:

https://github.com/coreos/fedora-coreos-tracker/issues/510

[travier]

With https://pagure.io/workstation-ostree-config/pull-request/306 & https://pagure.io/workstation-ostree-config/pull-request/303 we now have bootupd in F36 & F37; still not enabled by default, but that should let folks try it and report feedback so that we can fix issues.

[jvnknvlgl]

This might be relevant here: https://github.com/coreos/bootupd/issues/378

I'm having the same issue on 37, so I'm assuming bootupd cannot be used in Silverblue yet.

[travier]

Yes, we need to wait until this is fixed (and the package is updated).

[travier]

Should be mostly done now in Rawhide:

• https://github.com/coreos/bootupd/pull/387
• https://pagure.io/workstation-ostree-config/pull-request/307

I'll have to retro-actively write the Change Request.

[travier]

We still need to figure out the migration process before we remove ostree-grub2

[travier]

Automated migration would require: https://github.com/rhboot/shim/pull/502

[jmaibaum]

I have this issue trying to update the Secure boot dbx 77 -> 217 on Silverblue 36:

Blocked executable in the ESP, ensure grub and shim are up to date: /boot/efi/EFI/fedora/shimx64-fedora.efi Authenticode checksum [0ce02100f67c7ef85f4eed368f02bf7092380a3c23ca91fd7f19430d94b00c19] is present in dbx

I tried the workaround from https://github.com/fedora-silverblue/issue-tracker/issues/120#issuecomment-1177570662:

sudo rpm-ostree usroverlay 
wget https://kojipkgs.fedoraproject.org//packages/shim/15.6/1/x86_64/shim-x64-15.6-1.x86_64.rpm
sudo rpm -i  --reinstall shim-x64-15.6-1.x86_64.rpm```

Using shim-x64-15.6-2.x86_64.rpm instead of shim-x64-15.6-1.x86_64.rpm, as this was what rpm-ostree db list repored for my install. fwupdmgr upgrade still fails with the same error, but now I also get a secure boot signature violation, so I had to disable secure boot for now. Is this even a valid workaround for the Secure Boot dbx update failing?

https://bugzilla.redhat.com/show_bug.cgi?id=2127995 seems to suggest that there is a workaround, which sounds similar to what I tried above, but so far it doesn't work.

[jmaibaum]

I notice that the dbx update fails due to /boot/efi/EFI/fedora/shimx64-fedora.efi being present. This file is missing in /usr/lib/ostree-boot/efi/.... Would it help to remove the file?

[jmaibaum]

Moving /boot/efi/EFI/fedora/shimx64-fedora.efi helped getting the Secure Boot dbx update install successfully, yet I still can't reenable Secure Boot.

I am now trying to get bootupd to work:

$ bootupctl status
error: connecting to /run/bootupd.sock: ENOENT: No such file or directory

[jmaibaum]

Got past the previous error by systemctl enable bootupd.socket, now I am at

$ sudo bootupctl status
error: internal error: Could not find "/dev/disk/by-partlabel/EFI-SYSTEM"

And seing that https://github.com/coreos/bootupd/pull/387 is only released in bootupd 0.2.8, but Silverblue 36 only has

$ bootupctl update --help
bootupctl-update 0.2.6

I guess that this is as far as I can get for now?

[travier]

Working on the change for this issue in https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd

[0rzech]

@travier Is there any proper manual workaround for this we could apply before the change proposal you linked is implemented? Something that wouldn't require us to download files from the Internet.

[jmaibaum]

After rebasing to Fedora Silverblue 37, bootupd is now on v0.2.8, but I still can't continue:

$ sudo bootupctl status
No components installed.
Detected: EFI: unknown
Boot method: EFI
$ sudo bootupctl adopt-and-update
error: internal error: Component EFI has no available update

[travier]

I don't have a workaround that I've tested. Help welcomed. Maybe https://github.com/fedora-silverblue/issue-tracker/issues/120#issuecomment-1177515110 can be updated for F37 and might work but I have not tested this.

[jmaibaum]

I just retried https://github.com/fedora-silverblue/issue-tracker/issues/120#issuecomment-1177515110 like this:

sudo rpm-ostree usroverlay 
wget https://kojipkgs.fedoraproject.org//packages/shim/15.6/2/x86_64/shim-x64-15.6-2.x86_64.rpm
sudo rpm -i  --reinstall shim-x64-15.6-2.x86_64.rpm

Followed by a reboot, and then I tried to let bootupd take over again, but it's still not detecting anything in the EFI:

$ sudo bootupctl status
No components installed.
Detected: EFI: unknown
Boot method: EFI
$ sudo bootupctl adopt-and-update
error: internal error: Component EFI has no available update

Same as before.

[cgwalters]

Hi, using bootupd requires it to be enabled at build time, see https://pagure.io/workstation-ostree-config/pull-request/308

[travier]

Adding some more references here for cross-linking:

• https://pagure.io/workstation-ostree-config/pull-request/156
• https://github.com/ostreedev/ostree/pull/1929

[travier]

This has been accepted: https://pagure.io/fesco/issue/2900 Pending approval for rpm-ostree unified core change now: https://pagure.io/fesco/issue/2901

[travier]

Pushed back to F39

[secretmango]

A current workaround for this is:

if [ -d /sys/firmware/efi ]; then
  grub2-switch-to-blscfg
  grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
else
  block_device=$(lsblk -spnlo name $(grub2-probe --target=device /boot/grub2) | tail -n1)
  grub2-install $block_device
  touch /boot/grub2/.grub2-blscfg-supported
  grub2-mkconfig -o /boot/grub2/grub.cfg
fi

[juhp]

Pushed to F39

and F38?

[secretmango]

@juhp yes seems like a rather big problem with a small solution.

[travier]

https://github.com/coreos/fedora-coreos-tracker/issues/1441

[travier]

To clarify https://github.com/fedora-silverblue/issue-tracker/issues/120#issuecomment-1441638617: This has not landed yet because it's blocked by https://github.com/fedora-silverblue/issue-tracker/issues/333 which we had to report to F39.

[travier]

A current workaround for this is:

if [ -d /sys/firmware/efi ]; then
  grub2-switch-to-blscfg
  grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
else
  block_device=$(lsblk -spnlo name $(grub2-probe --target=device /boot/grub2) | tail -n1)
  grub2-install $block_device
  touch /boot/grub2/.grub2-blscfg-supported
  grub2-mkconfig -o /boot/grub2/grub.cfg
fi

This will only works if your bootloader is recent enough to have BLS support. Otherwise you'll get no boot entries.

[miabbott]

This will only works if your bootloader is recent enough to have BLS support. Otherwise you'll get no boot entries.

If someone runs this and finds they have no boot entries, what do the recovery steps look like?

[juhp]

Sorry to persist: so does "pushed to f39" mean deferred? Or do you still expect this to land in F38?

[miabbott]

Sorry to persist: so does "pushed to f39" mean deferred? Or do you still expect this to land in F38?

(If deferred, the Change pages should be updated to reflect that)

Deferred to F39.