Assertion error on AMD SEV-SNP snp-kernel-hashes-v3 patch with kernel-hashes=on

[youduda]

I am trying to run the snp-kernel-hashes-v3 patches of edk2 and qemu. I built them using the containers from kata-containers CCv0 (edk2 and qemu).

I am trying to boot the seL4 kernel with it, but it seems like there is something wrong.

/home/freund/qemu/bin/qemu-system-x86_64 \
-nodefaults -nographic \
-enable-kvm \
-cpu EPYC-v4 \
-machine pc-q35-7.1,vmport=off,confidential-guest-support=sev0 \
-object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,policy=0x30000,kernel-hashes=on \
-drive if=pflash,format=raw,unit=0,file=AMDSEV.fd,readonly=on \
-m size=1G \
-serial mon:stdio \
-kernel images/kernel-x86_64-pc99 \
-initrd images/capdl-loader-image-x86_64-pc99
qemu-system-x86_64: ../target/i386/sev.c:1541: snp_launch_update_kernel_hashes: Assertion `sev_snp->kernel_hashes_data' failed.

The same command with kernel-hashes=off results in:

BdsDxe: failed to load Boot0002 "Grub Bootloader" from Fv(7CB8BDC9-F8EB-4F34-AAEA-3EE4AF6516A1)/FvFile(B5AE312C-BC8A-43B1-9C62-EBB826DD5D07): Not Found
QEMU: Terminated

This boots as expected:

/home/freund/qemu/bin/qemu-system-x86_64 \
-nodefaults -nographic \
-enable-kvm \
-cpu EPYC-v4 \
-machine pc-q35-7.1,vmport=off \
-m size=1G \
-serial mon:stdio \
-kernel images/kernel-x86_64-pc99 \
-initrd images/capdl-loader-image-x86_64-pc99

Any idea what I am missing?

[tlendacky]

There have been some issues around the kernel-hashes support, because of an rng-seed issue, and around SNP. There were some patches recently posted upstream:

OVMF: https://listman.redhat.com/archives/edk2-devel-archive/2023-March/059923.html Qemu: https://lore.kernel.org/qemu-devel/20230302092347.1988853-1-dovmurik@linux.ibm.com/

You might have to use those with trees associated with the latest SNP hypervisor patches:

https://lore.kernel.org/lkml/20230220183847.59159-1-michael.roth@amd.com/

Give that a try and see if things work for you.

[youduda]

Thanks for the note. I am running the latest OVMF and Qemu, but not the latest hypervisor patches.

With the guest being a Linux kernel instead of the seL4 kernel the boot process with kernel-hashes=on works. Are there dependencies on the guest kernel for the boot itself?

[tlendacky]

Thanks for the note. I am running the latest OVMF and Qemu, but not the latest hypervisor patches.

With the guest being a Linux kernel instead of the seL4 kernel the boot process with kernel-hashes=on works. Are there dependencies on the guest kernel for the boot itself?

I wouldn't think so. Adding Dov (@dubek) to see if he has any insights.

[dubek]

The failed assertion complains about a NULL sev_snp->kernel_hashes_data. That pointer is filled in sev_add_kernel_loader_hashes() which is called from x86_load_linux() (in hw/i386/x86.c).

I don't know how seL4 boots -- maybe it doesn't go through x86_load_linux()?

[dubek]

Also I suggest adding -trace kvm_sev* to QEMU's command-line, maybe it'll reveal something about the flow (compare it between a linux kernel and a seL4 kernel).

[dubek]

Looking again at x86_load_linux(), I found two early returns from that function, which happen before sev_add_kernel_loader_hashes():

1. tries load_multiboot(...) in hw/i386/x86.c:851
2. tries pvh_enabled && load_elfboot(...) in hw/i386/x86.c:860

If either of these checks is true for the seL4 kernel file, then we won't reach sev_add_kernel_loader_hashes(); that could explain the difference you see between booting with seL4 kernel and a linux kernel.

@youduda can you please run file on both kernel files?

[youduda]

Sure, here is the output

$ file images/vmlinuz-linux
images/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 6.2.2-arch1-1 (linux@archlinux) #1 SMP PREEMPT_DYNAMIC Fri, 03 Mar 2023 15:58:31 +0000, RO-rootFS, swap_dev 0XB, Normal VGA
$ file images/kernel-x86_64-pc99
images/kernel-x86_64-pc99: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped

The trace output is similar:

kvm_sev_init type SEV-SNP flags 0x0
kvm_sev_snp_launch_start policy 0x30000 gosvw (null)
kvm_sev_change_state uninit -> launch-update
kvm_sev_snp_launch_update addr 0x7f09db000000 gpa 0xffc00000 len 0x400000 (Normal page)
kvm_sev_snp_launch_update addr 0x7f09dc600000 gpa 0x800000 len 0x9000 (Zero page)
kvm_sev_snp_launch_update addr 0x7f09dc60a000 gpa 0x80a000 len 0x3000 (Zero page)
kvm_sev_snp_launch_update addr 0x7f09dc60d000 gpa 0x80d000 len 0x1000 (Secrets page)
kvm_sev_snp_launch_update addr 0x7f09dc60e000 gpa 0x80e000 len 0x1000 (Cpuid page)
qemu-system-x86_64: ../target/i386/sev.c:1541: snp_launch_update_kernel_hashes: Assertion `sev_snp->kernel_hashes_data' failed.

kvm_sev_init type SEV-SNP flags 0x0
kvm_sev_snp_launch_start policy 0x30000 gosvw (null)
kvm_sev_change_state uninit -> launch-update
kvm_sev_snp_launch_update addr 0x7ff918a00000 gpa 0xffc00000 len 0x400000 (Normal page)
kvm_sev_snp_launch_update addr 0x7ff8d4600000 gpa 0x800000 len 0x9000 (Zero page)
kvm_sev_snp_launch_update addr 0x7ff8d460a000 gpa 0x80a000 len 0x3000 (Zero page)
kvm_sev_snp_launch_update addr 0x7ff8d460d000 gpa 0x80d000 len 0x1000 (Secrets page)
kvm_sev_snp_launch_update addr 0x7ff8d460e000 gpa 0x80e000 len 0x1000 (Cpuid page)
kvm_sev_snp_launch_update addr 0x7ff8d460f000 gpa 0x80f000 len 0x1000 (Normal page)
kvm_sev_snp_launch_update addr 0x7ff8d4610000 gpa 0x810000 len 0x10000 (Zero page)
kvm_sev_snp_launch_finish id_block (null) id_auth (null) host_data (null)
kvm_sev_change_state launch-update -> running

Edit: I added some tracing and it is load_multiboot(...) that prevents sev_add_kernel_loader_hashes().

[dubek]

Below is a patch with an attempt to add the kernel hashes support to load_multiboot(), but I'm not sure if it'll work: load_multiboot() doesn't set the fw_cfg entries FW_CFG_CMDLINE_DATA and FW_CFG_SETUP_DATA. I'm not sure how OVMF will behave if these entries don't exist. We might need to set them to empty buffers (or a single \0 string in case of cmdline).

diff --git a/hw/i386/multiboot.c b/hw/i386/multiboot.c
index 963e29362e..760b7a6b9c 100644
--- a/hw/i386/multiboot.c
+++ b/hw/i386/multiboot.c
@@ -31,6 +31,8 @@
 #include "elf.h"
 #include "sysemu/sysemu.h"
 #include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "target/i386/sev.h"

 /* Show multiboot debug output */
 //#define DEBUG_MULTIBOOT
@@ -164,6 +166,7 @@ int load_multiboot(X86MachineState *x86ms,
     uint32_t cmdline_len;
     GList *mods = NULL;
     g_autofree char *kcmdline = NULL;
+    SevKernelLoaderContext sev_load_ctx = {};

     /* Ok, let's see if it is a multiboot image.
        The header is 12x32bit long, so the latest entry may be 8192 - 48. */
@@ -396,11 +399,19 @@ int load_multiboot(X86MachineState *x86ms,
     fw_cfg_add_i32(fw_cfg, FW_CFG_KERNEL_SIZE, mbs.mb_buf_size);
     fw_cfg_add_bytes(fw_cfg, FW_CFG_KERNEL_DATA,
                      mbs.mb_buf, mbs.mb_buf_size);
+    sev_load_ctx.kernel_data = mbs.mb_buf;
+    sev_load_ctx.kernel_size = mbs.mb_buf_size;

     fw_cfg_add_i32(fw_cfg, FW_CFG_INITRD_ADDR, ADDR_MBI);
     fw_cfg_add_i32(fw_cfg, FW_CFG_INITRD_SIZE, sizeof(bootinfo));
     fw_cfg_add_bytes(fw_cfg, FW_CFG_INITRD_DATA, mb_bootinfo_data,
                      sizeof(bootinfo));
+    sev_load_ctx.initrd_data = (char *)mb_bootinfo_data;
+    sev_load_ctx.initrd_size = sizeof(bootinfo);
+
+    char *empty_str = g_strdup("");
+    sev_load_ctx.cmdline_data = empty_str;
+    sev_load_ctx.cmdline_size = 1;

     if (multiboot_dma_enabled) {
         option_rom[nb_option_roms].name = "multiboot_dma.bin";
@@ -410,5 +421,9 @@ int load_multiboot(X86MachineState *x86ms,
     option_rom[nb_option_roms].bootindex = 0;
     nb_option_roms++;

+    if (sev_enabled()) {
+        sev_add_kernel_loader_hashes(&sev_load_ctx, &error_fatal);
+    }
+
     return 1; /* yes, we are multiboot */
 }

[youduda]

Thanks a lot for the patch, this seems to improve the situation. The -trace kvm_sev* log output is now identical to the Linux kernel log.

The output is now the same as with kernel-hashes=off from the initial post above:

BdsDxe: failed to load Boot0002 "Grub Bootloader" from Fv(7CB8BDC9-F8EB-4F34-AAEA-3EE4AF6516A1)/FvFile(B5AE312C-BC8A-43B1-9C62-EBB826DD5D07): Not Found

[dubek]

OK, that error message means that OVMF verification of the kernel hashes failed, and it went on to boot from an embedded grub, which doesn't exist (that's fine). That's out stopgap that the guest doesn't start if the hashes don't match. If you have OVMF debug logs (port 0x402) captured, you can look for the word Verify or Verifier to see what's happening (and compared it to successful boot with linux).

[dubek]

The qemu option to capture the debug port is:

-global isa-debugcon.iobase=0x402 -debugcon file:ovmf.log 

This is what I get for successful verification in OVMF (linux kernel):

...
...
BlobVerifierLibSevHashesConstructor: Found injected hashes table in secure location
Select Item: 0x17
Select Item: 0x8
FetchBlob: loading 8304128 bytes for "kernel"
Select Item: 0x18
Select Item: 0x11
VerifyBlob: Found GUID 4DE79437-ABD2-427F-B835-D5B172D2045B in table
VerifyBlob: Hash comparison succeeded for "kernel"
Select Item: 0xB
FetchBlob: loading 105807810 bytes for "initrd"
Select Item: 0x12
VerifyBlob: Found GUID 44BAF731-3A2F-4BD7-9AF1-41E29169781D in table
VerifyBlob: Hash comparison succeeded for "initrd"
Select Item: 0x14
FetchBlob: loading 83 bytes for "cmdline"
Select Item: 0x15
VerifyBlob: Found GUID 97D02DD8-BD20-4C94-AA78-E7714D36AB2A in table
VerifyBlob: Hash comparison succeeded for "cmdline"
...
...

[youduda]

Well, you already said the cmdline could be an issue and indeed it is:

BlobVerifierLibSevHashesConstructor: Found injected hashes table in secure location
Select Item: 0x17
Select Item: 0x8
FetchBlob: loading 10244096 bytes for "kernel"
Select Item: 0x18
Select Item: 0x11
VerifyBlob: Found GUID 4DE79437-ABD2-427F-B835-D5B172D2045B in table
VerifyBlob: Hash comparison succeeded for "kernel"
Select Item: 0xB
FetchBlob: loading 88 bytes for "initrd"
Select Item: 0x12
VerifyBlob: Found GUID 44BAF731-3A2F-4BD7-9AF1-41E29169781D in table
VerifyBlob: Hash comparison succeeded for "initrd"
Select Item: 0x14
VerifyBlob: Found GUID 97D02DD8-BD20-4C94-AA78-E7714D36AB2A in table
VerifyBlob: Hash comparison failed for "cmdline"
Error: Image at 0003E6F5000 start failed: Access Denied

I added at ovmf an ignore for the cmdline just to check if it would work.

if (CompareGuid (&mSevCmdlineHashGuid, Guid)) {
  return EFI_SUCCESS;
}

But it still goes into grub. This looks like there is something wrong:

[Security] 3rd party image[0] can be loaded after EndOfDxe: VenMedia(1428F772-B64A-441E-B8C3-9EBDD7F893C7)/kernel.
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 3E696040
QemuLoadKernelImage: LoadImage(): Unsupported

Full log: ovmf.log

[dubek]

But it still goes into grub. This looks like there is something wrong:

[Security] 3rd party image[0] can be loaded after EndOfDxe: VenMedia(1428F772-B64A-441E-B8C3-9EBDD7F893C7)/kernel.
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 3E696040
QemuLoadKernelImage: LoadImage(): Unsupported

It might be that the AmdSev OVMF package requires the kernel to have an EFI stub. When building linux, this means setting CONFIG_EFI_STUB=y. Not sure if there's an equivalent for seL4.

Does your kernel boot with "normal" OVMF (OvmfPkg/OvmfPkgX64.dsc) ? (you'd have to set kernel-hashes=off)

[youduda]

Ok thanks for the hint. It doesn't boot with OvmfPkgX64 as well. I'll dig into this and then update the issue accordingly or close it. For now, thanks for all your effort!