Open buckaroogeek opened 2 years ago
As far as I can tell from the UEFI specification, the firmware is supposed to support FAT16 or FAT32:
13.3.1.1 File System Format
The EFI firmware must support the FAT32, FAT16, and FAT12 variants of the EFI file system. What variant of EFI FAT to use is defined by the size of the media. The rules defining the relationship between media size and FAT variants is defined in the specification for the EFI file system.
We seem to not be specifying the actual FAT size when running mkfs.fat
, which looks like it makes it also auto-detect the right one based on size:
-F FAT-SIZE
Specifies the type of file allocation tables used (12, 16 or 32 bit). If nothing is speci‐
fied, mkfs.fat will automatically select between 12, 16 and 32 bit, whatever fits better for
the filesystem size.
which seems to fit with the UEFI definition.
AFAIK, we've had other users successfully boot FCOS on the Raspberry Pi 4. I suspect this may be related to the fact that you're booting via USB instead of SD card? @dustymabe Have you tried booting via USB?
My understanding is that the ESP on a non-removable disk must be FAT32. UEFI spec 2.9 section 13.3 says:
EFI encompasses the use of FAT32 for a system partition, and FAT12 or FAT16 for removable media.
We had the same bug in Container Linux (https://github.com/coreos/bugs/issues/2246).
Thanks for the information about the specification and updates. I should spell check before I submit! I recall that during my initial efforts to install and run FCOS on RPi4 hardware last February (see https://forums.raspberrypi.com/viewtopic.php?t=304318 for a write up if interested), this was not an issue I ran into. The USB disk I uses is a KINGSTON SA400S37120G (120 GB). Please let me know if I can provide any more information and/or testing.
@dustymabe Have you tried booting via USB?
I've been recently testing things via both uboot and UEFI (working on documentation to add to our docs site). I have to admit I have seen some cases where the pi4 will loop on trying partition 0 like @buckaroogeek suggested, but I've also taken that exact USB out and plugged it back in and it worked the next time, which was odd.
Maybe use of FAT16 makes the issue more likely to happen? Maybe I'm misremembering? I definitely know things are working on an SD card just fine the way it is today.
I checked the eeprom firmware page on github (https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/release-notes.md) and see this note: "Update the FAT detection to support FAT16 for EFI/ESD paritions with GPT instead of assuming FAT32. The latest firmware is also required for a similar update." This is for a 4 Oct update.
This is a corner case that likely affects more than just the Pi, so we should fix it regardless. Fix in https://github.com/coreos/coreos-assembler/pull/2491.
And thanks for the report!
Glad to help.
Going to re-open until the change lands
There's a complication. Microsoft's EFI FAT32 specification says:
The FAT type—one of FAT12, FAT16, or FAT32—is determined by the count of clusters on the volume and nothing else.
FAT32 is defined to have at least 65525 clusters, and with one 4 KiB sector/cluster, that means the ESP on a 4Kn disk needs to be at least 257 MiB. Otherwise EDK II (and presumably other UEFI firmware) won't accept it as a valid filesystem. Our ESP is 127 MiB, and the partition layout is difficult to change because it's also wired into Butane.
We could switch to FAT32 only in the 512b image, which would improve the situation somewhat, but would create a weird inconsistency between our image types. Or we could change the partition layout again, or stick with FAT16.
If we switched the ESP to FAT32, we should probably update the Butane RAID template to do the same. Butane doesn't have any way to know the boot disk's sector size (the user may not know either), so effectively it can't use different templates for 512b and 4Kn disks.
discuss at meeting?
Yeah. Added meeting label.
Was curious about what Anaconda did here. It uses blivet, which looks like it currently always creates a FAT16 filesystem: https://github.com/storaged-project/blivet/blob/314675891c86ba185deabc52f77ca6389973d9a1/blivet/formats/fs.py#L925-L926
* AGREED: For now, we will document workarounds for systems that can't
boot from a FAT16 ESP, including older Raspberry Pi firmware. Next
time we change the partition layout, we will switch the ESP to FAT32
in both the 512b and 4Kn images, selecting a size of at least 257 MB
to fit a valid FAT32 filesystem.
My thanks for the thorough review of this issue. Documentation as noted by @lucab will be helpful and useful. Shall I close or wait for the documentation?
I've filed https://github.com/coreos/fedora-coreos-docs/issues/326 to track the docs issue. Let's leave this open, though, since we'd still like to switch to FAT32 in the future.
For the record, there's a lot of good discussion in the meeting logs. pjones pointed out that we're not violating the UEFI spec here because the spec isn't binding on us; we're just depending on firmware behavior which is apparently not mandated by the spec.
@dustymabe pointed out that enlarging the ESP implicates #586. We addressed that issue by recommending a minimum root filesystem size, which doesn't help if the same-size rootfs is placed after a larger ESP.
For at least the RPi4 part of this I added a section to the docs in https://github.com/coreos/fedora-coreos-docs/pull/336 that talks about how to update the EEPROM. The docs URL is: https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-raspberry-pi4/#_updating_eeprom_on_raspberry_pi_4
This intersects with the UKI bits - with UKIs, we don't need a separate /boot
partition, so we could use that space at least to expand the ESP.
Bug
Raspberry Pi 4 will not boot Fedora CoreOS if the EFI-System partition is FAT16. Convert to FAT32 and the RPi4 boots. The coreos-installer appears to use FAT16 for the EFI system partition. I could not see an option to change this at coreos-installer run time.
Host Operating System Version
Fedora 34
Target Operating System Version
Fedora CoreOS 34
coreos-installer Version
coreos-installer-0.10.0-2.fc34.x86_64
Expected Behavior
Use coreos-installer to create a bootable USB mass storage device for Raspberry Pi 4. CoreOS should boot once coreos-installer completes without error and the USB device is plugged into the RPi4 and powered on.
Actual Behavior
FCOS will not boot on the RPi4 until the EFI-System partition is reformatted as FAT32. It appears that coreos-installer formats the partition as FAT16. The RPi4 displays a message: Trying partition 0. Trying to read partition as fat. (with other text), This message block repeats blocking the boot process.
Reproduction Steps
Other Information
I am using v 1.31 of the pftp/uefi firmware. I can manually reformat the partition to FAT32 and FCOS boots as expected.