Oak Containers: QEMU not recognizing SEV-SNP: `parameter 'qom-type' does not accept value 'sev-snp-guest'`

smherwig commented 1 month ago

I'm working on the main branch (commit c5190c07aa8da7436760f39bb5de8e27afb44844).

I'm attempting to run the rust_hello_world_trusted_bundle container using the following command:

sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_launcher \
    --system-image oak_containers/system_image/target/image.tar.xz \
    --container-bundle target/rust_hello_world_trusted_bundle.tar \
    --vmm-binary $(which qemu-system-x86_64) \
    --stage0-binary generated/stage0_bin \
    --kernel oak_containers/kernel/target/bzImage \
    --initrd target/stage1.cpio \
    --ramdrive-size 5000000 \
    --vm-type sev-snp

Running this command results in the following log lines:

[2024-10-11T20:18:40Z INFO  oak_containers_launcher] Launcher service listening on port 43977
[2024-10-11T20:18:40Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=" "-chardev" "socket,id=consock,fd=11" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 43977,hostfwd=tcp:127.0.0.1:41161-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:45071-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=1748300,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:43977", kill_on_drop: true }
qemu-system-x86_64: -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=: Parameter 'qom-type' does not accept value 'sev-snp-guest'

The last log line indicates a QEMU issue.

Our setup is:

server: Supermicro CloudDC A+ Server AS -2015CS-TNR
cpu: 1x EPYC 7354P 32C/64T CPU. This is an EPYC 4th Generation AMD Genoa chip, and supports SEV-SNP.
kernel: 6.11.0-rc3-snp-host-85ef1ac03941. We followed AMD's Using SEV with AMD EPYC Processors guide to setup the BIOS, SEV firmware, and host kernel. Specifically, we're running AMD's kernel as per the instructions on AMD's sep-latest branch
nix: We're using the nix-2.14.1 setup as described in docs/development.md .

Additionally, I manually had to sudo modprobe vhost_vsock.

Thanks. I appreciated any help.

conradgrobler commented 1 month ago

I believe that AMD SEV-SNP support has not been upstreamed for QEMU yet, so the version of QEMU that is installed in the nix environment will not work with it. To use SEV-SNP I think you will have to install the version from the AMD repository and use that version in the launcher.

smherwig commented 1 month ago

To further clarify, we are using the nix install of qemu:

$ which qemu-system-x86_64
/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64

$ qemu-system-x86_64 --version
QEMU emulator version 9.0.1
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

conradgrobler commented 1 month ago

I think that you will have to build QEMU from https://github.com/AMDESE/qemu/tree/snp-latest to support AMD SEV-SNP rather than use the nix version

smherwig commented 1 month ago

@conradgrobler: thank you; that worked after applying @dingelish's QEMU patch dingelish/qemu@876e262 to also correct the vmw_vsock_virtio_transport error at launch.

dingelish commented 1 month ago

@conradgrobler we verified the stack on the latest upstream stable 6.11 kernel and upstream qemu. AMD's branch is bit of old.

dingelish commented 1 month ago

@smherwig could you please do me a favor? I want to know if you can run the vanilla qemu without that patch after applying this to the oak_containers_launcher

diff --git a/oak_containers/launcher/src/qemu.rs b/oak_containers/launcher/src/qemu.rs
index 487d46865..71890ac2d 100644
--- a/oak_containers/launcher/src/qemu.rs
+++ b/oak_containers/launcher/src/qemu.rs
@@ -168,7 +168,7 @@ impl Qemu {
             params.memory_size.clone().unwrap_or("8G".to_string())
         );
         // SEV's feature configuration.
-        let sev_config_object = "id=sev0,cbitpos=51,reduced-phys-bits=1";
+        let sev_config_object = "id=sev0,cbitpos=51,reduced-phys-bits=2";
         // TDX machine suffix
         let tdx_machine_suffix = ",kernel_irqchip=split,memory-encryption=tdx,memory-backend=ram1";
         let tdx_common_object = format!(

also this patch

diff --git a/oak_containers/launcher/src/qemu.rs b/oak_containers/launcher/src/qemu.rs
index 487d46865..71890ac2d 100644
--- a/oak_containers/launcher/src/qemu.rs
+++ b/oak_containers/launcher/src/qemu.rs
@@ -168,7 +168,7 @@ impl Qemu {
             params.memory_size.clone().unwrap_or("8G".to_string())
         );
         // SEV's feature configuration.
-        let sev_config_object = "id=sev0,cbitpos=51,reduced-phys-bits=1";
+        let sev_config_object = "id=sev0,cbitpos=51";
         // TDX machine suffix
         let tdx_machine_suffix = ",kernel_irqchip=split,memory-encryption=tdx,memory-backend=ram1";
         let tdx_common_object = format!(

smherwig commented 1 month ago

@dingelish - Thanks. For either patch, I still get the QEMU error:

Parameter 'qom-type' does not accept value 'sev-snp-guest'

when using vanilla QEMU v9.0.1 (/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64).

Patch 1 Trial Run

$ sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_hello_world_untrusted_app  \
    --system-image oak_containers/system_image/target/image.tar.xz  \
    --container-bundle target/rust_hello_world_trusted_bundle.tar \
    --vmm-binary $(which qemu-system-x86_64) \
    --stage0-binary generated/stage0_bin  \
    --kernel oak_containers/kernel/target/bzImage  \
    --initrd target/stage1.cpio  \
    --ramdrive-size 5000000  \
    --vm-type sev-snp rest
ARGS: Args { server_type: Rest, launcher_args: Args { system_image: "oak_containers/system_image/target/image.tar.xz", container_bundle: "target/rust_hello_world_trusted_bundle.tar", application_config: [], qemu_params: Params { vmm_binary: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64", stage0_binary: "generated/stage0_bin", kernel: "oak_containers/kernel/target/bzImage", initrd: "target/stage1.cpio", memory_size: Some("8G"), num_cpus: 1, ramdrive_size: 5000000, telnet_console: None, virtio_guest_cid: None, pci_passthrough: None, vm_type: SevSnp }, communication_channel: Network } }
SERVER ADDR Ok(0.0.0.0:8006)
[2024-10-16T20:06:59Z INFO  oak_containers_launcher] Launcher service listening on port 34419
[2024-10-16T20:06:59Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-m" "8G" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=2,id-auth=" "-chardev" "socket,id=consock,fd=12" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 34419,hostfwd=tcp:127.0.0.1:37327-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:34865-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=145077,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:34419", kill_on_drop: true }
qemu-system-x86_64: -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=2,id-auth=: Parameter 'qom-type' does not accept value 'sev-snp-guest'

Patch 2 Trial Run

$ sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_hello_world_untrusted_app \
    --system-image oak_containers/system_image/target/image.tar.xz  \
    --container-bundle target/rust_hello_world_trusted_bundle.tar  \
    --vmm-binary $(which qemu-system-x86_64) \
    --stage0-binary generated/stage0_bin  \
    --kernel oak_containers/kernel/target/bzImage \
    --initrd target/stage1.cpio  \
    --ramdrive-size 5000000  \
    --vm-type sev-snp rest
ARGS: Args { server_type: Rest, launcher_args: Args { system_image: "oak_containers/system_image/target/image.tar.xz", container_bundle: "target/rust_hello_world_trusted_bundle.tar", application_config: [], qemu_params: Params { vmm_binary: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64", stage0_binary: "generated/stage0_bin", kernel: "oak_containers/kernel/target/bzImage", initrd: "target/stage1.cpio", memory_size: Some("8G"), num_cpus: 1, ramdrive_size: 5000000, telnet_console: None, virtio_guest_cid: None, pci_passthrough: None, vm_type: SevSnp }, communication_channel: Network } }
SERVER ADDR Ok(0.0.0.0:8006)
[2024-10-16T20:11:37Z INFO  oak_containers_launcher] Launcher service listening on port 37075
[2024-10-16T20:11:37Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-m" "8G" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,id-auth=" "-chardev" "socket,id=consock,fd=12" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 37075,hostfwd=tcp:127.0.0.1:41391-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:42761-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=149753,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:37075", kill_on_drop: true }
qemu-system-x86_64: -object sev-snp-guest,id=sev0,cbitpos=51,id-auth=: Parameter 'qom-type' does not accept value 'sev-snp-guest'

smherwig commented 1 month ago

@dingelish - It also occurred to me that by "vanilla" you meant the https://github.com/AMDESE/qemu/tree/snp-latest version (without your QEMU patch dingelish/qemu@876e262).

When I try using that version, I get the following errors:

Patch 1 Trial Run

$ sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_hello_world_untrusted_app   \
    --system-image oak_containers/system_image/target/image.tar.xz  \
    --container-bundle target/rust_hello_world_trusted_bundle.tar  \
    --vmm-binary /home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64  \
    --stage0-binary generated/stage0_bin  \
    --kernel oak_containers/kernel/target/bzImage  \
    --initrd target/stage1.cpio  \
    --ramdrive-size 5000000  \
    --vm-type sev-snp rest
ARGS: Args { server_type: Rest, launcher_args: Args { system_image: "oak_containers/system_image/target/image.tar.xz", container_bundle: "target/rust_hello_world_trusted_bundle.tar", application_config: [], qemu_params: Params { vmm_binary: "/home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64", stage0_binary: "generated/stage0_bin", kernel: "oak_containers/kernel/target/bzImage", initrd: "target/stage1.cpio", memory_size: Some("8G"), num_cpus: 1, ramdrive_size: 5000000, telnet_console: None, virtio_guest_cid: None, pci_passthrough: None, vm_type: SevSnp }, communication_channel: Network } }
SERVER ADDR Ok(0.0.0.0:8006)
[2024-10-16T20:22:59Z INFO  oak_containers_launcher] Launcher service listening on port 33725
[2024-10-16T20:22:59Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-m" "8G" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=2,id-auth=" "-chardev" "socket,id=consock,fd=12" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 33725,hostfwd=tcp:127.0.0.1:39829-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:40269-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=155995,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:33725", kill_on_drop: true }
stage0 INFO: starting...
stage0 DEBUG: Using fw_cfg to create the E820 table
stage0 DEBUG: early E820 entry: [0x0000000000000000-0x0000000000080000), len 524288, type RAM
stage0 DEBUG: early E820 entry: [0x0000000000080000-0x00000000000a0000), len 131072, type ACPI
stage0 DEBUG: early E820 entry: [0x00000000000f0000-0x0000000000100000), len 65536, type RESERVED
stage0 DEBUG: early E820 entry: [0x0000000000100000-0x00000000c0000000), len 3220176896, type RAM
stage0 DEBUG: early E820 entry: [0x00000000feffc000-0x00000000ff000000), len 16384, type RESERVED
stage0 DEBUG: early E820 entry: [0x0000000100000000-0x0000000240000000), len 5368709120, type RAM
stage0 INFO: Enabled SEV features: SevStatus(SEV_ENABLED | SEV_ES_ENABLED | SNP_ACTIVE)
stage0 INFO: starting SEV-SNP memory validation
stage0 INFO: SEV-SNP memory validation complete.
stage0 INFO:   Validated using 2 MiB pages: 0
stage0 INFO:   Validated using 4 KiB pages: 2096912
stage0 INFO:   Valid state not updated: 0
stage0 INFO:   RMP page size mismatch errors (fallback to 4K): 4095
stage0 DEBUG: Kernel cmdline:  console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:33725
stage0 DEBUG: Kernel image size 5726208
stage0 DEBUG: Kernel image start address 0x0000000002000000
stage0 DEBUG: Kernel entry point 0x0000000002000200
stage0 INFO: Using x2APIC for AP initialization.
stage0 DEBUG: Local APIC: ProcessorLocalApic { header: ControllerHeader { structure_type: 0, len: 8 }, processor_uid: 0, apic_id: 0, flags: LocalApicFlags(ENABLED) }
stage0 DEBUG: uninteresting structure: 1
stage0 DEBUG: uninteresting structure: 1
stage0 DEBUG: uninteresting structure: 4
stage0 INFO: Expected number of APs: 0, started number of APs: 0
stage0 DEBUG: Initial RAM disk size 2802132
stage0 DEBUG: Initial RAM disk address 0x000000003fd53000
stage0 ERROR: panicked at stage0/src/paging.rs:254:5:
assertion failed: page_start < Size2MiB::SIZE
^C

Patch 2 Trial Run

$ sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_hello_world_untrusted_app \
    --system-image oak_containers/system_image/target/image.tar.xz  \
    --container-bundle target/rust_hello_world_trusted_bundle.tar \
    --vmm-binary /home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64  \
    --stage0-binary generated/stage0_bin  \
    --kernel oak_containers/kernel/target/bzImage  \
    --initrd target/stage1.cpio  \
    --ramdrive-size 5000000 \
    --vm-type sev-snp rest
ARGS: Args { server_type: Rest, launcher_args: Args { system_image: "oak_containers/system_image/target/image.tar.xz", container_bundle: "target/rust_hello_world_trusted_bundle.tar", application_config: [], qemu_params: Params { vmm_binary: "/home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64", stage0_binary: "generated/stage0_bin", kernel: "oak_containers/kernel/target/bzImage", initrd: "target/stage1.cpio", memory_size: Some("8G"), num_cpus: 1, ramdrive_size: 5000000, telnet_console: None, virtio_guest_cid: None, pci_passthrough: None, vm_type: SevSnp }, communication_channel: Network } }
SERVER ADDR Ok(0.0.0.0:8006)
[2024-10-16T20:21:05Z INFO  oak_containers_launcher] Launcher service listening on port 41409
[2024-10-16T20:21:05Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-m" "8G" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,id-auth=" "-chardev" "socket,id=consock,fd=12" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 41409,hostfwd=tcp:127.0.0.1:42245-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:41525-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=151994,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:41409", kill_on_drop: true }
qemu-system-x86_64: -object sev-snp-guest,id=sev0,cbitpos=51,id-auth=: Parameter 'reduced-phys-bits' is missing

dingelish commented 1 week ago

@smherwig I upgraded my workstation to use the upstream linux kernel from kernel.org, and upstream 9.1.1 qemu. they worked together perfectly without any patch. I'm using debian 12 bookworm on that machine.

project-oak / oak