search

OSInside / kiwi

KIWI - Appliance Builder Next Generation
https://osinside.github.io/kiwi
GNU General Public License v3.0
300 stars 152 forks source link

ISO Boot Failure on Supermicro Servers in UEFI Mode #2512

Open zohourih opened 6 months ago

zohourih commented 6 months ago

Problem description

ISOs generated by kiwi-ng with firmware="efi" are not detected as bootable in UEFI mode when mounted over IPMI on Supermicro servers, and it is impossible to boot from them. The same ISO boots fine in Legacy mode. The same ISO boots fine on HPE and DELL servers, but Supermicro informed us that is due to an extra emulation layer they have implemented for the virtual CDROM. Standard Linux ISOs from Ubuntu/SLES/etc. boot fine on Supermicro servers in both UEFI and Legacy mode. The issue is reproducible on all generations of Supermicro servers.

Per Supermicro, the structure of the ISOs generated by kiwi-ng is missing certain components that are required for correct boot in UEFI mode. A similar problem existed in Mint 21.2 ISO which was corrected in Mint 21.3 ISO. The following information is provided to us by Supermicro:

1) Mint 21.2 – Cannot boot up in UEFI mode through IPMI Virtual CDROM on X12DDW-A6.
2) Mint 21.3 – Boot up fine in UEFI mode through IPMI Virtual CDROM.

Mint 21.2 UEFI in X12DDW-A6: No FS device.

image

Compare the two ISOs: Mint 21.2 doesn’t have the efi.img file, that’s the file for EFI boot information. Mint fixed the issue in ISO 21.3

image

Per Supermicro, the following steps are required to correct images generated by kiwi-ng for correct UEFI boot (copied verbatim from Supermicro's response):

Original ISO:

image

Modified ISO:

image

Modification steps:

1) Create an efiboot.img for EFI boot 2) added isolinux.bin and libraries for Legacy boot.

Generate new efiboot.img (I changed from 2880 to 5760 so bootx64.efi can fit in):

dd if=/dev/zero of=EFI/BOOT/efiboot.img bs=512 count=5760

Format the boot image with F 12:

mkfs.msdos -F 12 -n 'KLA-BOOT-ISO' EFI/BOOT/efiboot.img

Create Directory structure for the efiboot image:

mmd -i EFI/BOOT/efiboot.img ::EFI
mmd -i EFI/BOOT/efiboot.img ::EFI/BOOT
mcopy -i EFI/BOOT/efiboot.img EFI/BOOT/bootx64.efi ::EFI/BOOT/bootx64.efi

Generate new ISO using xorriso to include EFI platform support and EFI boot image:

xorriso -as mkisofs -no-emul-boot -boot-load-size 4 -boot-info-table -iso-level 4 -b isolinux/isolinux.bin -eltorito-alt-boot -eltorito-platform 0xEF -volid "KLA-BOOT-ISO" -eltorito-boot EFI/BOOT/efiboot.img -no-emul-boot -o /tmp/klaboot.iso isoinfo .

isohybrid -u /tmp/klaboot.iso:

image

image

image

Is there some configuration we can change in config.xml or kiwi config.yml that would allow us to build ISOs that follow the structure mentioned above?

Expected behaviour

ISOs created using kiwi-ng with firmware="efi" should be detected and bootable in UEFI mode when mounted on Supermicro servers over IPMI.

Steps to reproduce the behaviour

  1. Download the config file below and replace *server* with target repository. config.zip
  2. Use the following kiwi config file and build the ISO: 
    xz:
- options: '--threads=8'
container:
- compress: xz
mapper:
- part_mapper: kpartx
  3. Procure any Supermicro X11 or newer server that supports UEFI boot, and set boot mode to UEFI in the BIOS.
  4. Mount the ISO over IPMI and try to boot from it.
  5. The UEFI virtual CDROM will never show up among boot options as the ISO is not detected as bootable in UEFI mode.

OS and Software information

schaefi commented 6 months ago

Thanks much for the very detailed and professional report :+1:

As a side note, we dropped support for isolinux (see #2439 for details). Thus legacy boot is handled through grub for quite some time. As we are talking about EFI boot this information is JFI and I only mentioned it because your fix procedure added isolinux.bin and friends for the legacy boot. The isolinux project is EOL since some time.

but Supermicro informed us that is due to an extra emulation layer

Yeah every system that has EFI boot capabilities and allows to boot from iso9660 requires a FAT based loader structure as part of the ISO layer. We do that in kiwi too which makes me wonder why it did not boot on Supermicro. I looked at one of our integration tests to see the differences. For example: https://build.opensuse.org/package/live_build_log/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-live:Secure/images/x86_64

In there if you search for mcopy you find the instructions:

[   71s] [ DEBUG   ]: 19:18:15 | EXEC: [qemu-img create /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz 20M]
[   71s] [ DEBUG   ]: 19:18:15 | EXEC: [mkdosfs -n BOOT /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz]
[   71s] [ DEBUG   ]: 19:18:15 | EXEC: [mcopy -Do -s -i /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz /usr/src/packages/KIWI-iso/live-media.caxzpjnr/EFI ::]

The FAT image efi-loader.2w3kejdz is a temporary file used in the later xorriso call and not put as a file on the ISO. In your case you created this as efiboot.img and placed it also as a file to the ISO filesystem.

Further down the log you can search for xorriso and you will find this

[  157s] [ DEBUG   ]: 19:19:40 | EXEC: [/usr/bin/xorriso -application_id 0x6be39c0e -publisher SUSE LINUX GmbH -preparer_id KIWI - https://github.com/OSInside/kiwi -volid CDROM -joliet on -padding 0 -outdev /usr/src/packages/KIWI-iso/kiwi-test-image-live.x86_64-1.42.3.iso -map /usr/src/packages/KIWI-iso/live-media.caxzpjnr / -chmod 0755 / -- -boot_image grub bin_path=boot/x86_64/loader/eltorito.img -boot_image grub grub2_mbr=/usr/src/packages/KIWI-iso/live-media.caxzpjnr/boot/x86_64/loader/boot_hybrid.img -boot_image grub grub2_boot_info=on -boot_image any partition_offset=16 -boot_image any cat_path=boot/x86_64/boot.catalog -boot_image any cat_hidden=on -boot_image any boot_info_table=on -boot_image any platform_id=0x00 -boot_image any emul_type=no_emulation -boot_image any load_size=2048 -append_partition 2 0xef /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz -boot_image any next -boot_image any efi_path=--interval:appended_partition_2:all:: -boot_image any platform_id=0xef -boot_image any emul_type=no_emulation]

As you can see the FAT image is used as an any loader of the form

-boot_image any load_size=2048 -append_partition 2 0xef /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz

Which I believe is very similar to your -eltorito-boot instructions

Ok, so from that perspective I have several questions:

  1. I see your FAT image is created as a FAT12 filesystem. Our procedure creates a FAT16 (default) filesystem. Is that an issue for Supermicro ?
  2. Can you please compare your xorriso build instructions with the provided kiwi log and share your thoughts if you agree with me that kiwi basically does the same thing ?
  3. You placed your FAT image file as EFI/BOOT/efiboot.img into the ISO filesystem. Is this a requirement for Supermicro ? From my perspective it's not needed to place the FAT image as a file to the ISO, kiwi does not do this.

Would be great if you can respond to my findings. Currently I only think that the FAT format could be an issue which things are not booting on Supermicro. I don't have this hardware at hand so I'm afraid I can't test it.

Thanks

zohourih commented 6 months ago

@schaefi Thank you for looking into this. I am afraid apart from the problem statement part, all the technical information I have put in the issue were provided by the Supermicro engineer who worked on the case, and, unfortunately, I don't have the answer to your questions. I will reach out to the Supermicro engineer and see if they are willing to participate in this discussion.

The original feedback I got from Supermicro was that unless the isohybrid test succeeds on the ISO, it won't boot using Supermicro's virtual CDROM over UEFI. The test certainly fails on the original ISO, but succeeds on the modified ISO. I am not quite sure what the success criteria is in that case, and it might be possible to pass that criteria in ways other than the one suggested by Supermicro. I think it could also be valuable to study the Linux Mint 21.2 to 21.3 ISO change, as well, and determine why that change was implemented, as this is a very recent case.

I have access to several Supermicro servers and if there is anything in particular you want to be tested, I can make the change and rebuild the ISO and test and report the results.

zohourih commented 6 months ago

@schaefi I got more information from Supermicro about this case. The person who provided the instructions to correct the ISO said they followed the steps detailed here and studied the format of Rocky Linux ISO and replicated the same format. Here is a verbatim quote regarding their requirement:

To boot ISO in EFI using ATEN virtual CD, then the ISO should include an efiboot.img (it can be named for any name) that contains the EFI boot directory “/EFI/BOOT” and boot images “bootx64.efi and grubx64.efi” which you can find in all standard Linux ISOs.

Is there any standard Linux ISO (Ubuntu, SLES, Rocky, etc.) that follows the same EFI boot format as the ISOs generated by kiwi? If there is such standard ISO, it should fails to boot using Supermicro's virtual CDROM in UEFI mode, and we might then be able to compel them to resolve the problem on their side; otherwise, they are not going to allocate resources to resolve a problem that doesn't occur with standard Linux ISOs. It is worth noting that we prefer if we can find a solution at kiwi level since, even if Supermicro fixes the problem on their side, we will need to update the BIOS/BMC FW of thousands of servers scattered across several countries to be able to get the fix, which is not practical.

schaefi commented 6 months ago

There is no standard in designing an ISO layout, it's just a filesystem with data. However, there is a clear EFI standard and that has to be followed in order to boot via EFI. Of course kiwi follows this as all others do too. For booting an ISO via EFI an embedded FAT image is required because the EFI standard requires a FAT signature plus EFI binaries below EFI/BOOT. kiwi also does that as shown in the integration test log. At this point, even though there is a standard, many hardware vendors already deviate and add stuff to change the behavior.

I'm happy to help and fix what is needed to boot on supermicro. I also believe that it cannot be a big issue because technically kiwi does what is required to boot an ISO from EFI. I'm afraid but the comparison to others does not help here because the design for an ISO layout from a content perspective varies between the distributions and is also not an issue as long as the technical requirements to boot are met.

Thus it's required to understand the real cause of the boot issue. In order to find that we need to clarify on the open questions that I still have:

We can only continue with a test and patch when we have clarity on the above questions and I believe this can only be done together with a supermicro engineer contributing to this thread. That's because I have a number of devices here on my table but no supermicro system ;)

Thanks

zohourih commented 6 months ago

@schaefi I did actually ask the person who worked on the case to participate in this thread, the response I got was that he is not a "build engineer" and only followed the steps I mentioned earlier to "fix" the ISO we provided them with (albeit, there is no proof the ISO was broken to begin with). He is not sure what the boot requirements of Supermicro's UEFI is, either. It is quite difficult to get Supermicro to co-operate on matters that involve custom solutions, and as I said, unless the issue also happens with ISOs from a standard distribution, they are not going to engage their BIOS/UEFI engineers.

Nonetheless, I will pass your questions to Supermicro and see if they can provide an answer. I can predict with high confidence that the problem is exactly what you raised in your last question: Supermicro's bootloader probably cannot read the EFI image from the ISO header, and only looks for it inside the ISO filesystem.

zohourih commented 6 months ago

Verbatim copy of Supermicro's response:

1) As I told you before, I just follow the link from this github to modify the ISO image who happened to use FAT 12  (Fat12 is for floppy image).  I compared your ISO versus the standard Rocky Linux and Mint Linux before I did the modification.  
https://github.com/syzdek/efibootiso

2) Is FAT16 work?  I don’t know because I couldn’t find any information regarding ISO and FAT16.
https://www.file-recovery.com/recovery-understanding-file-system-fat.htm

3) I believe you can put the efiboot.img in any directory you want as long as you specified in the directory structure where to read the bootx64.efi in the ISO
# mmd -i EFI/BOOT/efiboot.img ::EFI
# mmd -i EFI/BOOT/efiboot.img ::EFI/BOOT
# mcopy -i EFI/BOOT/efiboot.img EFI/BOOT/bootx64.efi ::EFI/BOOT/bootx64.efi

4) Xorriso ISO image creator
https://unix.stackexchange.com/questions/283846/why-can-my-custom-build-iso-not-boot-in-efi
https://forums.centos.org/viewtopic.php?t=78987

FYI, we have been using the same BMC ATEN virtual media for ISO mount in all supermicro products for years (X9, X10,X11,X12,X13 ..) This is the first time I learned about the EFI ISO boot issue.  Mint Linux fixed their 21.2 ISO boot EFI issue with the release of 21.3 ISO if you want to compare between the two.

@schaefi Do you know of any standard distribution that embeds the EFI image in the ISO header, rather than including it as part of the ISO file system? Looks like Linux Mint was one, but they went back to putting the image in the ISO filesystem in 21.3 and the changelog of the 21.3 release does not actually mention why they did that. I am pretty sure this was not to restore compatibility with Supermicro's virtual CDROM, though (😆), and the 21.2 structure probably worked everywhere else as is. Unfortunately, Supermicro is going to refer to this case until the end of time, as "proof" that this is the only correct structure, even though we know this is not the case, and unfortunately, there is no official standard or spec to prove them wrong.

zohourih commented 5 months ago

@schaefi I checked several standard Linux ISOs to see if I can find any that doesn't work on Supermicro's servers. The Linux Mint 21.2 ISO that doesn't boot on Supermicro's servers in UEFI mode does actually have an efi.img file, but it is under /boot/grub/. In 21.3 there are two different efi.img files in the ISO: one under /boot/grub/, and one directly in the root of the ISO. I am not sure why they did that. Other difference between the two ISOs is that the 21.2 ISO fails the isohybrid test, but 21.3 passes.

On the other hand, the Ubuntu 22.04 ISOs neither pass the isohybrid test, nor have any efi.img file anywhere inside the ISO, yet they boot fine on Supermicro's servers. I guess this means the problem is not actually related to whether efi.img is in the ISO header or inside the ISO, as there are working ISOs with both structures.

The remaining point is regarding FAT 12 vs. FAT 16. Is there any way I can confirm if the EFI partition is FAT12 or FAT16 in an existing ISO such as Ubuntu's ISO?

Conan-Kudo commented 2 months ago

Did you try using firmware="uefi" instead? That tells kiwi to pull in all the normal resources from the distribution (e.g. shim and friends) and set up a more "conventional" UEFI ESP that you'd see from distro images.

Additionally, firmware="efi" means that if your system has a verified boot process using the Secure Boot feature, it will fail. firmware="uefi" is required for compatibility with Secure Boot.

zohourih commented 2 months ago

@Conan-Kudo Yes, we have tried "uefi", as well. Results are the same. We are not using secure boot in our environment.