Changes to enable measured direct boot and run SEV-SNP guests correctly?

[gianlucascopelliti]

Hi,

I would like to try out measured direct boot in order to include the measurements of kernel, initrd and kernel arguments to the attestation report of a SEV-SNP guest. So far I've been experimenting from the snp-latest branch but without success.

As far as I understand (e.g., from https://github.com/AMDESE/AMDSEV/issues/93#issuecomment-1514158728), to enable this we first need to build OVMF from OvmfPkg/AmdSev/AmdSevX64.dsc. This generates a single OVMF.fd file instead of two separate OVMF_CODE.fd and OVMF_VARS.fd files.

Now, what are the next steps to do? In particular, what would we need to change in launch_qemu.sh / qemu in order to set things up correctly? My question is not only related to the OVMF.fd file itself, but more in general to all the qemu arguments that would differ compared to the "normal" way (i.e., without measured direct boot).

Thanks!

[tlendacky]

You would need to update the launch_qemu.sh script to specify only a single "-drive if=pflash,..." argument specifying the OVMF.fd file. Then you would need to add the -kernel, -initrd and -append arguments to boot everything directly using the specified kernel, initrd and command line arguments. Additionally, you need to "kernel-hashes=on" to the sev-snp-guest object definition.

I don't run this very often, so I think that is all that is needed, but YMMV.

[gianlucascopelliti]

Indeed, this works, thanks! I had tried this previously but I could not boot the guest due to some error, but now I have rebuilt QEMU and OVMF and it's working.

Side note: when using -kernel and -initrd it could not boot the guest properly but went to recovery mode (initramfs). To solve this issue, I had to run the guest normally (without -kernel) and run sudo update-initramfs -u. After that, trying direct boot again worked correctly.

[gianlucascopelliti]

Sorry, actually I was mistaken, it's still not working properly.

I found out that if I add kernel-hashes=on it always boots in recovery mode and I'm basically stuck. Instead, using kernel-hashes=off boots normally.

Below is the full QEMU command:

./usr/local/bin/qemu-system-x86_64 -enable-kvm -cpu EPYC-v4,host-phys-bits=true -smp 4 -machine type=q35,confidential-guest-support=sev0,memory-backend=ram1,kvm-type=protected,vmport=off -object memory-backend-memfd-private,id=ram1,size=4G,share=true -object sev-snp-guest,id=sev0,policy=0x30000,cbitpos=51,reduced-phys-bits=1,init-flags=0,host-data=b2l3bmNvd3FuY21wbXA,kernel-hashes=on -drive if=pflash,format=raw,unit=0,file=/home/ecaiogs/sev/AMDSEV/snp-release-2023-10-25/usr/local/share/qemu/OVMF.fd,readonly=on -drive file=/home/ecaiogs/sev/sevsnp.qcow2,if=none,id=disk0,format=qcow2 -device virtio-scsi-pci,id=scsi0,disable-legacy=on,iommu_platform=true -device scsi-hd,drive=disk0 -nographic -kernel /home/ecaiogs/sev/vmlinuz-6.6.0-rc1-snp-guest-f86c98f8b43d -append "console=ttyS0 earlyprintk=serial root=/dev/sda2" -initrd /home/ecaiogs/sev/initrd.img-6.6.0-rc1-snp-guest-f86c98f8b43d -monitor pty -monitor unix:monitor,server,nowait -netdev type=tap,script=no,downscript=no,id=net0,ifname=tap6 -device virtio-net-pci,mac=52:54:00:c8:91:06,netdev=net0,disable-legacy=on,iommu_platform=true,romfile=

The working command is exactly the same except for the kernel-hashes parameter that is set to off.

This is the error from the recovery mode:

Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... [    2.527022] Btrfs loaded, zoned=yes, fsverity=yes
Scanning for Btrfs filesystems
done.
Begin: Will now check root file system ... fsck from util-linux 2.37.2
[/usr/sbin/fsck.ext4 (1) -- /dev/sda2] fsck.ext4 -a -C0 /dev/sda2 
/dev/sda2: recovering journal
/dev/sda2: clean, 310/114688 files, 81872/458752 blocks
done.
[    2.586656] EXT4-fs (sda2): mounted filesystem f63fde81-0384-4c4e-96b1-dcc0a31f8241 ro with ordered data mode. Quota mode: none.
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /dev on /root/dev failed: No such file or directory
mkdir: can't create directory '/root/lib/': Read-only file system
Warning: No /lib/modules in target. cannot help.
done.
mount: mounting /run on /root/run failed: No such file or directory
run-init: can't execute '/sbin/init': No such file or directory
Target filesystem doesn't have requested /sbin/init.
run-init: can't execute '/sbin/init': No such file or directory
run-init: can't execute '/etc/init': No such file or directory
run-init: can't execute '/bin/init': No such file or directory
run-init: can't execute '/bin/sh': No such file or directory
run-init: can't execute '': No such file or directory
No init found. Try passing init= bootarg.

BusyBox v1.30.1 (Ubuntu 1:1.30.1-7ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) [    2.905673] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3

Any advice?

[tlendacky]

There should be some output from OVMF. Look for messages prefixed with BlobVerifierLibSevHashesConstructor and/or VerifyBlob.

But it looks like for some reason it just can't find the root drive.

[dubek]

When kernel-hashes=off I'd expect OVMF (if built as AmdSevX64) to not allow booting the kernel at all. A look into the OVMF logs per @tlendacky 's suggestion might be a good start.

Also try running qemu with tracing of kvm_sev_* logs to see if it reveals anything.

[gianlucascopelliti]

Thanks for your responses. I tried again (rebuilding the kernel, QEMU and OVMF just to be sure) and I confirm what I wrote last time. Note that I built again OVMF from AmdSevX64.dsc:

• With kernel-hashes=off, it just boots normally without verifying the kernel. However, I noticed that it doesn't boot the kernel passed as input, but the usual one installed in the VM image.
• With kernel-hashes=on, OVMF verifies the hashes correctly, and then correctly boots the kernel passed as input (I verified with uname -r). From here, however, it goes in recovery mode (indeed it seems like it can't find the root drive).

Below is the output from the hash verification done by OVMF:

BlobVerifierLibSevHashesConstructor: Found injected hashes table in secure location
Select Item: 0x17
Select Item: 0x8
FetchBlob: loading 11533440 bytes for "kernel"
Select Item: 0x18
Select Item: 0x11
VerifyBlob: Found GUID 4DE79437-ABD2-427F-B835-D5B172D2045B in table
VerifyBlob: Hash comparison succeeded for "kernel"
Select Item: 0xB
FetchBlob: loading 111096191 bytes for "initrd"
Select Item: 0x12
VerifyBlob: Found GUID 44BAF731-3A2F-4BD7-9AF1-41E29169781D in table
VerifyBlob: Hash comparison succeeded for "initrd"
Select Item: 0x14
FetchBlob: loading 48 bytes for "cmdline"
Select Item: 0x15
VerifyBlob: Found GUID 97D02DD8-BD20-4C94-AA78-E7714D36AB2A in table
VerifyBlob: Hash comparison succeeded for "cmdline"

Tracing kvm_sev_* did not reveal anything interesting, just the logs below:

kvm_sev_init type SEV-SNP flags 0x0
qemu-system-x86_64: warning: Restricted memory (UPM) enabled, disabling SMM. Memory discard mode: none
kvm_sev_snp_launch_start policy 0x30000 gosvw (null)
kvm_sev_change_state uninit -> launch-update
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 17, size: 80000000, flags: 1
qemu-system-x86_64: warning: creating ROM device with private memory.
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 23, size: 400000, flags: 1
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 25, size: 20000, flags: 0
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 27, size: 20000, flags: 0
kvm_sev_snp_launch_update addr 0x7f4e4b000000 gpa 0xffc00000 len 0x400000 (Normal page)
kvm_sev_snp_launch_update addr 0x7f4e4c600000 gpa 0x800000 len 0x9000 (Zero page)
kvm_sev_snp_launch_update addr 0x7f4e4c60a000 gpa 0x80a000 len 0x3000 (Zero page)
kvm_sev_snp_launch_update addr 0x7f4e4c60d000 gpa 0x80d000 len 0x1000 (Secrets page)
kvm_sev_snp_launch_update addr 0x7f4e4c60e000 gpa 0x80e000 len 0x1000 (Cpuid page)
kvm_sev_snp_launch_update addr 0x7f4e4c60f000 gpa 0x80f000 len 0x1000 (Normal page)
kvm_sev_snp_launch_update addr 0x7f4e4c610000 gpa 0x810000 len 0x10000 (Zero page)
kvm_sev_snp_launch_finish id_block (null) id_auth (null) host_data (null)
kvm_sev_change_state launch-update -> running

I'll try to figure out why it doesn't boot properly. Will keep you updated.

[gianlucascopelliti]

I was finally able to boot properly using a fresh disk image (created with debootstrap) and passing root=/dev/sda to the kernel arguments.

I still haven't figured out why my previous image isn't booting properly, but it seems obvious now that it's not a problem related to SEV-SNP. Hence, I'll close this issue.

[enuoCM]

I was finally able to boot properly using a fresh disk image (created with debootstrap) and passing root=/dev/sda to the kernel arguments.

@gianlucascopelliti I'm facing the same problem. With kernel-hashes=on, the booted vm doesn't has /dev/sev-guest device, but dmesg | grep -i -e rmp -e sev output seems fine:

[    0.550548] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP
[    0.682529] SEV: APIC: wakeup_secondary_cpu() replaced with wakeup_cpu_via_vmgexit()
[    0.757173] SEV: Using SNP CPUID table, 31 entries present.
[    1.197467] SEV: SNP guest platform device initialized.

Do you have this issue?

[gianlucascopelliti]

Hi @enuoCM,

Uhm no, I did not have this issue. From the dmesg output I have an additional line that you don't have:

[    4.917243] sev-guest sev-guest: Initialized SEV guest driver (using vmpck_id 0)

I assume you are using the latest SNP-enabled kernel built from the snp-latest branch?