increase size of our /boot partition for new installs

[dustymabe]

Right now our /boot partition size is 384M. We have decided that we want to increase our /boot partition size because we have been hitting out of space issues.

We also believe that we should limit this change to newly deployed systems (i.e. don't try to re-arrange the partition table of updating systems).

In this ticket we should discuss:

• how large we should make the partition in the future
• how to prevent data loss when re-provisioning systems using coreos-installer
• how to handle existing Ignition configs that have partitioning directives
• ...

[bgilbert]

While we're at it, we could also resize the ESP (on 4Kn disks, or everywhere) to fix https://github.com/coreos/fedora-coreos-tracker/issues/993#issuecomment-941749104. We currently have to use FAT16 at least on the 4Kn ESP, which isn't standards-compliant.

[dustymabe]

While we're at it, we could also resize the ESP (on 4Kn disks, or everywhere) to fix #993 (comment). We currently have to use FAT16 at least on the 4Kn ESP, which isn't standards-compliant.

Also of note is https://fedoraproject.org/wiki/Changes/BiggerESP

[travier]

For UKIs, they will live in /boot AFAIK, just like we have the BLS entries there right now.

We could move everything to the ESP but that would mean using a different partition layout for BIOS & UEFI setups or do the change on first boot. We could merge the EFI & boot partitions on first boot for UEFI setups to get more space and move everything to the ESP. But would that be worth the complexity increase to skip a bootloader indirection layer and get 512 MB back?

Not sure about what we want to do.

In terms of space, we already need 3 times sizeof(kernel + initramfs) space for updates right now and we'll need the same for UKIs.

[dustymabe]

We discussed this in the community meeting today.

13:07:15*  dustymabe | #info For now the proposal is a 512M ESP and 1G /boot partition
                     | but we are following discussions about UKI to determine if that
                     | will change where (ESP or /boot) we put the large files
                     | (kernel+initramfs) in the future.

We also discussed briefly the potential challenges. From what I understand they are:

1. update the butane templates
2. update coreos-installer to be aware and not clobber data
   1. detect that it's overwriting an existing CoreOS installation
   2. compare the new image's partition table to the existing one
   3. fail if we're about to clobber a non-OS partition

It was determined that we don't think there is significant work related to people's existing Ignition configurations other than just communicating to them they may need updating. In the case where people are reprovisioning root everything gets moved around anyway. In the case where they aren't and a boundary would be violated Ignition will error. We just need to make people aware of the error and the necessary fix. We decided that writing a Fedora change proposal for this change would be a good idea to socialize it more widely.

[dustymabe]

The proposal made by @keszybz on the devel list (when discussing the BiggerESP change proposal) seems interesting:

If we want to change the default here, let's do some proper cleanup:

1. the split between ESP and XBOOTLDR is only useful in the case where ESP already existed and was small. If the installer is creating an ESP, it should just make it large enough.
2. having a second partition with a second (different) file system implementation just increases the footprint and attack surface for no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT in almost all realistic scenarios).
3. if there are bootloaders that don't read one or the other partition as they should, fix them.

Then we can make the ESP 1 GB and save space compared to current defaults.

[travier]

Another suggestion from Benjamin is to merge /boot and the ESP and have the BIOS bootloader chainload the ESP bootloader directly instead of having both BIOS & EFI bootloaders chainload the /boot bootloader like we have right now. Then we would need ostree support for vfat (https://github.com/ostreedev/ostree/issues/1951).

We would thus only loose the space used by the BIOS GRUB partition and everything else would be in the combined EFI/boot partition.

Also related to https://github.com/ostreedev/ostree/issues/1719 & https://github.com/ostreedev/ostree/issues/2753.

[dustymabe]

We discussed this in the community meeting today.

Specifically we discussed the side-proposal mentioned in https://github.com/coreos/fedora-coreos-tracker/issues/1465#issuecomment-1543245937. It seems to have some merit, but would need https://github.com/ostreedev/ostree/issues/1951 (as mentioned by @travier above) and we'd also need to determine if all architectures could pick up the change (combined ESP and /boot) without issue (i.e. does petitboot support VFAT?).

We may try to send a mail to the mailing list to see if the side-proposal could gain steam.

[bgilbert]

xref https://github.com/coreos/fedora-coreos-tracker/issues/1353, which requests a larger reserved partition in the aarch64 image.

[cgwalters]

xref https://github.com/ostreedev/ostree/issues/3066

[ericcurtin]

While I'm normally in favour of less partitions, I'm not 100% sure about merging /boot an /boot/efi partitions... I was thinking of using fs-verity to verify this new initoverlayfs thing we are working on, which I am placing in /boot as of today with vfat I'm not sure if there are alternate verification techniques that can be used there. But am open to ideas...

In automotive at the moment, boot time is a focus so every millisecond matters. I know ext4 also has pretty decent boot metrics, so this is also worth considering as a factor. vfat is even simpler, so maybe it's even faster (or slower?), I genuinely don't know.

https://www.linkedin.com/posts/sprado_linux-embeddedlinux-filesystems-activity-7087731117813899264-YWkV?utm_source=share&utm_medium=member_desktop

https://elinux.org/images/0/02/Opdenacker-finding-best-block-filesystem.pdf

https://www.youtube.com/watch?v=kLxM4FyiVpQ&ab_channel=TheLinuxFoundation

[ericcurtin]

Although nevermind, thinking about this more, we can have just a vfat partition, if we need functionality from a more advanced filesystem like ext4/erofs , we can just loopback mount a file with that filesystem contained.

So scratch what I said before, one single vfat partition actually does make sense to me as long as it can be mounted faster or at an equivalent rate than an ext4 filesystem on boot.

[fabiendupont]

Interesting read that proposes a partition scheme: https://0pointer.net/blog/fitting-everything-together.html. The article covers a number of topics, each one bringing more info about the partitioning scheme. It is summarized in the "Partitions Setup" section.