Issues found while integrating Clover with QEMU

[DaanDeMeyer]

I'm trying to get rid of all the BIOS logic in mkosi by emulating UEFI on BIOS using Clover. mkosi is an image builder that produces bootable disk images. My WIP PR can be found here. The commit that adds clover is this one.

I managed to get a bootable image that chainloads systemd-boot in QEMU's SeaBIOS using Clover, but with a few caveats:

• Using qemu's if=virtio option to load the disk image makes qemu hang on boot when using Clover
• Using qemu's -cpu max option leads to a CPU exception on boot in qemu when using Clover
• Using qemu's -nographic option also makes qemu hang on boot when using Clover

These issues can be reproduced by cloning the PR branch, and running the following command from within the mkosi repository root directory:

# This will successfully boot in qemu
sudo bin/mkosi --bootable --boot-protocols bios -f qemu
# This will hang on boot in qemu
sudo bin/mkosi --bootable --boot-protocols bios --qemu-headless -f qemu

If you undo the -cpu max or if=virtio changes in the run_qemu_cmdline() function, qemu will also fail in the ways described above.

Let me know if I should come up with an easier reproducer. If needed I can attach a generated disk image along with a qemu cmdline to boot it as well.

A picture of the CPU exception when booting Clover in SeaBIOS with -cpu max:

When qemu hangs on boot when using -nographic, the following output is shown before qemu hangs:

SeaBIOS (version 1.16.0-1.fc36)
Booting from Hard Disk...
6

qemu also starts busy looping on its assigned CPU core when this happens

[SergeySlice]

clone the Clover into your local disk. cd CloverBootloader\Qemu ./launch See how Clover works in Qemu

[DaanDeMeyer]

Yes, that would work, because the launch script doesn't use any of the options I mentioned above. If I look at the file, it seems to use these options:

  "$qemu_path" \
    -L "$(dirname "$qemu_path")" \
    -m 2048 \
    -cpu Penryn \
    -bios "$bios_file" \
    -machine q35 \
    -device ahci,id=ahi \
    -drive format=raw,id=hda,file=./disk_image_gpt.img \
    -usb \
    -device usb-mouse,bus=usb-bus.0,port=2 \
    -device usb-kbd,bus=usb-bus.0,port=1 \
    -serial file:./serial0_qemu_out.log \
    -gdb tcp::9000

I can make Clover work in qemu as well if I don't use if=virtio, -cpu max or -nographic options. Those options are useful though, virtio disks are much more performant than the default disk type in qemu, cpu max is required to boot centos kernels properly and -nographic is required to make qemu work in a terminal.

So I would like it if Clover could be made to work in qemu even if if=virtio, -cpu max or -nographic are used.

[SergeySlice]

Why -cpu max? There is Penryn as best solution.

[DaanDeMeyer]

Penryn won't work as many Linux distributions are looking to switch to x86-64-v2 which Penryn doesn't support. Ideally Clover would work regardless of what CPU type is selected in qemu

[SergeySlice]

Make debug.log in Clover to see if there is a problem in Clover itself. ` Boot

Debug `

[DaanDeMeyer]

I enabled Debug in my config.plist but there is no debug log in EFI/CLOVER/misc after boot, even when I do a successful boot without using -cpu max, -nographic or if=virtio.

My config.plist can be found here: https://github.com/systemd/mkosi/pull/1051/commits/22209fe687729177417817f0af4c831e8db3dcc9

[DaanDeMeyer]

I built Clover locally with -D DEBUG_ON_SERIAL_PORT and that actually fixed the boot when -nographic is used. So whatever DEBUG_ON_SERIAL_PORT changes fixes qemu boot when -nographic is used.

It'd be great if the serial console could be enabled by default in clover without enabling extra debugging, but I also wouldn't mind if it was able to boot without a serial console when no graphics device is available.

[DaanDeMeyer]

Serial console debug logs when booting with -cpu max (qemu hangs after this, likely because of the CPU exception):

SeaBIOS (version 1.16.0-1.fc36)
Booting from Hard Disk...
0:100  0:100  MemLog inited, TSC freq: 2111979549
0:100  0:000  CPU was calibrated with ACPI PM Timer
0:100  0:000  LegacyRegion2: Chipset/proc: 0x29C08086
0:100  0:000   Intel Series 4 and similar (PAM 0x90-0x96)
0:101  0:000   Test PAM1=(0x41=00, 0x5a=00, 0x81=00, 0x91=11) at chipset 29C08086
0:101  0:000  , Install = Success
0:144  0:043  BiosVideoDriverBindingStart!
0:144  0:000  BiosVideoCheckForVbe
0:145  0:000  EdidOverride not found
0:145  0:000   Edid1+
0:146  0:000     0 640x480 attr=BB - ok, 640x480, working, highest, pref=0
0:146  0:000     1 800x600 attr=BB - ok, 800x600, working, highest, pref=1
0:147  0:000     2 1024x768 attr=BB - ok, 1024x768, working, highest, pref=2
0:148  0:000     3 1280x1024 attr=BB - ok
0:148  0:000     4 1600x1200 attr=BB - ok
0:148  0:000     5 1152x864 attr=BB - ok
0:149  0:000     6 1280x768 attr=BB - ok
0:149  0:000     7 1280x800 attr=BB - ok
0:150  0:000     8 1280x960 attr=BB - ok
0:150  0:000     9 1440x900 attr=BB - ok
0:151  0:000    10 1400x1050 attr=BB - ok
0:151  0:000    11 1680x1050 attr=BB - ok
0:152  0:000    12 1280x720 attr=BB - ok
0:152  0:000    13 1920x1080 attr=BB - ok
0:155  0:002    14 640x480 attr=BB - ok, 640x480, working
0:156  0:000    15 800x600 attr=BB - ok, 800x600, working
0:157  0:000    16 1024x768 attr=BB - ok, 1024x768, working, highest, pref=16
0:158  0:000    17 1280x1024 attr=BB - ok
0:159  0:001    18 1600x1200 attr=BB - ok
0:160  0:001    19 1152x864 attr=BB - ok
0:160  0:000    20 1280x768 attr=BB - ok
0:161  0:000    21 1280x800 attr=BB - ok
0:162  0:000    22 1280x960 attr=BB - ok
0:162  0:000    23 1440x900 attr=BB - ok
0:163  0:000    24 1400x1050 attr=BB - ok
0:163  0:000    25 1680x1050 attr=BB - ok
0:164  0:000    26 1920x1200 attr=BB - ok
0:165  0:000    27 1280x720 attr=BB - ok
0:165  0:000    28 1920x1080 attr=BB - ok
0:166  0:000    29 1600x900 attr=BB - ok
0:167  0:000    30 1600x900 attr=BB - ok
0:167  0:000  BV new mode: 16 1024x768
0:169  0:001   SetMode pref 16 (16) = Success
0:511  0:341  CloverX64 : Image base = 0x3E198000
0:511  0:000  Clover ImageHandle = 3f0d4a18
0:513  0:001  === [ Starting Clover ] =========================
0:513  0:000  Now is 16.07.2022,  12:11:40 (GMT)
0:513  0:000  Starting Clover revision: 5148 (master, commit a5e78463a) on CLOVER EFI
0:514  0:000  Build id: 20220711230911-a5e7846-dirty
0:515  0:000  Build with: [Args: -D DEBUG_ON_SERIAL_PORT -b DEBUG | --cmd-len=50000 -D DEBUG_ON_SERIAL_PORT --conf=/home/daandemeyer/projects/CloverBootloader/Conf -D USE_LOW_EBDA -a X64 -b DEBUG -t GCC53 -n 9 | OS: Fedora 36]
0:516  0:001  === [ Get Smbios ] ==============================
0:517  0:000  Type 16 Index = 0
0:517  0:000  Total Memory Slots Count = 1
0:517  0:000  Type 17 Index = 0
0:518  0:000  Ignoring insane frequency value 0MHz
0:518  0:000  SmbiosTable.Type17->SlotIndex = 0
0:519  0:000  SmbiosTable.Type17->ModuleSize = 1024MB
0:519  0:000  SmbiosTable.Type17->Type = 0
0:519  0:000  SmbiosTable.Type17->Frequency = 0MHz
0:520  0:000  SmbiosTable.Type17->Bank/Device =  DIMM 0
0:520  0:000  SmbiosTable.Type17->Vendor = QEMU
0:521  0:000  SmbiosTable.Type17->SerialNo = 
0:521  0:000  SmbiosTable.Type17->PartNo = 
0:522  0:000  Boot status=0
0:522  0:000  Running on: 'Standard PC (Q35 + ICH9, 2009)' with board ''
0:522  0:000  === [ GetCPUProperties ] ========================
0:523  0:000  CPU Vendor = 756E6547 Model=806EC
0:523  0:000   The CPU supported SSE4.1
0:523  0:000   The CPU supported RDRAND
0:524  0:000  BrandString = Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz
0:524  0:000   The CPU not supported turbo
0:525  0:000   Energy PerfBias is  not visible:
6

[DaanDeMeyer]

Serial console debug logs when booting with if=virtio (qemu hangs after this):

SeaBIOS (version 1.16.0-1.fc36)
Booting from Hard Disk...
0:100  0:100  MemLog inited, TSC freq: 2111981369
0:100  0:000  CPU was calibrated with ACPI PM Timer
0:100  0:000  LegacyRegion2: Chipset/proc: 0x29C08086
0:100  0:000   Intel Series 4 and similar (PAM 0x90-0x96)
0:101  0:000   Test PAM1=(0x41=00, 0x5a=00, 0x81=00, 0x91=11) at chipset 29C08086
0:101  0:000  , Install = Success
0:144  0:043  BiosVideoDriverBindingStart!
0:145  0:000  BiosVideoCheckForVbe
0:145  0:000  EdidOverride not found
0:145  0:000   Edid1+
0:146  0:000     0 640x480 attr=BB - ok, 640x480, working, highest, pref=0
0:146  0:000     1 800x600 attr=BB - ok, 800x600, working, highest, pref=1
0:147  0:000     2 1024x768 attr=BB - ok, 1024x768, working, highest, pref=2
0:148  0:000     3 1280x1024 attr=BB - ok
0:148  0:000     4 1600x1200 attr=BB - ok
0:148  0:000     5 1152x864 attr=BB - ok
0:149  0:000     6 1280x768 attr=BB - ok
0:150  0:000     7 1280x800 attr=BB - ok
0:150  0:000     8 1280x960 attr=BB - ok
0:150  0:000     9 1440x900 attr=BB - ok
0:151  0:000    10 1400x1050 attr=BB - ok
0:151  0:000    11 1680x1050 attr=BB - ok
0:152  0:000    12 1280x720 attr=BB - ok
0:152  0:000    13 1920x1080 attr=BB - ok
0:155  0:002    14 640x480 attr=BB - ok, 640x480, working
0:156  0:000    15 800x600 attr=BB - ok, 800x600, working
0:156  0:000    16 1024x768 attr=BB - ok, 1024x768, working, highest, pref=16
0:157  0:000    17 1280x1024 attr=BB - ok
0:158  0:000    18 1600x1200 attr=BB - ok
0:160  0:001    19 1152x864 attr=BB - ok
0:160  0:000    20 1280x768 attr=BB - ok
0:161  0:000    21 1280x800 attr=BB - ok
0:161  0:000    22 1280x960 attr=BB - ok
0:162  0:000    23 1440x900 attr=BB - ok
0:163  0:000    24 1400x1050 attr=BB - ok
0:163  0:000    25 1680x1050 attr=BB - ok
0:164  0:000    26 1920x1200 attr=BB - ok
0:165  0:000    27 1280x720 attr=BB - ok
0:165  0:000    28 1920x1080 attr=BB - ok
0:166  0:000    29 1600x900 attr=BB - ok
0:166  0:000    30 1600x900 attr=BB - ok
0:167  0:000  BV new mode: 16 1024x768
0:169  0:002   SetMode pref 16 (16) = Success

[SergeySlice]

Hang with -cpu=max is easy. QEMU emalates CoffeLake CPU but don't emulate MSR for energy bias. So crash. When CPU=Penryn the MSR is non asked. About virtio I don't know what is this and how it works.

[DaanDeMeyer]

Hang with -cpu=max is easy. QEMU emalates CoffeLake CPU but don't emulate MSR for energy bias. So crash.

Thanks for the hint! I found /sys/modules/kvm/parameters/ignore_msrs which instructs KVM to not crash on MSRs that are not emulated. Enabling that fixes booting with -cpu max.

About virtio I don't know what is this and how it works.

For virtio, it's a custom protocol that seems to need custom support in the bootloader? I found implementations for OVMF (See the Virtio folders here) and SeaBIOS (see the virtio files here).

I guess Clover would require custom support for the virtio protocol as well?

[Randrianasulu]

A picture of the CPU exception when booting Clover in SeaBIOS with -cpu max:

Similar screen appear if I try to boot efi/bios usb stick prepared via ArchWiki article on two machines with real CPUs :

one is AMD FX4300 CPU on Asrock 760gm-hdv - no real UEFI here. Another machine is HP ProBook 4730s with i5-2450m (also somewhat strange UEFI? it exist in bios settings, but can't boot from usb stick I formatted as pure UEFI setup)

[Randrianasulu]

hm, strange enough building on Linux with cross gcc 11.4.0 and clang++ 17.0.6 (hopefully) creates bootable on qemu with -enable-kvm and -cpu host iso

for some reason main buildme script demands xcode for creating ISO yet I can run manually

guest@slax:~/botva/src/src/CloverBootloader/CloverPackage$ ./makeiso

and it will work. No nice graphical menu but at least textual interface show up.

I modded

BaseTools/Conf/tools_def.template

for my specific crosscompiler in /opt/kgcc64 (c-only for kernel builds)

and

BaseTools/Source/C/Makefiles/header.makefile

else
 BUILD_CFLAGS = -MD -fshort-wchar -fno-strict-aliasing -fwrapv \    
 --fno-delete-null-pointer-checks -Wall -Werror \    
 +-fno-delete-null-pointer-checks -Wall \^M       
 -Wno-deprecated-declarations -Wno-stringop-truncation -Wno-restrict \   -Wno-unused-result -nostdlib -g                                                                    endif

because Werror there was breaking compile in brotli headers for me (vla error)