SEV-SNP init and guest launch fail, error 0x9003, rc -5

[Zauney]

Hi,

We are trying to execute SEV-SNP on a dedicated host on AWS.

To get started, we followed the instructions as stated in the readme of the snp-latest branch. Unfortunately we encountered some issues during the boot of the host machine and when starting a VM with enabled SNP:

Boot of host machine (SEV init seems to fail):

ubuntu@XXXXX:~$ sudo dmesg | egrep "SEV|sev|RMP|rmp|SNP|snp|ccp"
[    0.000000] Linux version 6.5.0-rc2-snp-host-XXXXX (ubuntu@XXXXX) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #2 SMP Sat Sep  9 18:14:35 UTC 2023
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.5.0-rc2-snp-host-XXXXX root=PARTUUID=YYYYY ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1
[    0.840661] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.5.0-rc2-snp-host-XXXXX root=PARTUUID=YYYYY ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1
[    0.840721] Unknown kernel command line parameters "BOOT_IMAGE=/boot/vmlinuz-6.5.0-rc2-snp-host-XXXXX", will be passed to user space.
[    3.693304] AMD-Vi: SNP enabled
[    3.693431] SEV-SNP: RMP table physical address [0x000000601ff00000 - 0x00000060e07fffff]
[    4.301763] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP
[    4.302309] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP
[    4.302849] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP
[    4.303392] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP
[    4.303934] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP
[    4.304474] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP
[    4.305015] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP
[    4.305556] AMD-Vi: Extended features (0x841f77e022094ace, 0x0): PPR X2APIC NX IA GA PC SNP
[    6.577884]     BOOT_IMAGE=/boot/vmlinuz-6.5.0-rc2-snp-host-XXXXX
[    7.839205] usb usb1: Manufacturer: Linux 6.5.0-rc2-snp-host-XXXXX xhci-hcd
[    7.846561] usb usb2: Manufacturer: Linux 6.5.0-rc2-snp-host-XXXXX xhci-hcd
[    7.856569] usb usb3: Manufacturer: Linux 6.5.0-rc2-snp-host-XXXXX xhci-hcd
[    7.862522] usb usb4: Manufacturer: Linux 6.5.0-rc2-snp-host-XXXXX xhci-hcd
[    7.882985] ccp 0000:54:00.1: no command queues available
[    7.883535] ccp 0000:54:00.1: sev enabled
[    7.883537] ccp 0000:54:00.1: psp enabled
[    7.884705] ccp 0000:c0:00.1: no command queues available
[    7.884738] ccp 0000:c0:00.1: psp enabled
[    7.959650] ccp 0000:54:00.1: SEV firmware update successful
[   18.930947] ccp 0000:54:00.1: SEV: failed to INIT error 0x9003, rc -5
[   18.931030] ccp 0000:54:00.1: SEV-SNP API:1.54 build:1
[   18.942095] kvm_amd: SEV-ES and SEV-SNP supported: 509 ASIDs
[   18.942097] kvm_amd: SEV enabled (ASIDs 510 - 509)
[   18.942098] kvm_amd: SEV-ES enabled (ASIDs 1 - 509)
ubuntu@XXXXX:~$ uname -a
Linux XXXXX 6.5.0-rc2-snp-host-XXXXX #2 SMP Sat Sep  9 18:14:35 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Start of Guest VM (sev_kvm_init fails):

ubuntu@XXXXX:~/AMDSEV$ sudo ./launch-qemu.sh -hda /home/ubuntu/AMDSEV/Image.qcow2 -sev-snp
32+0 records in
1+0 records out
512 bytes copied, 0.000227072 s, 2.3 MB/s
/home/ubuntu/AMDSEV/usr/local/bin/qemu-system-x86_64 -enable-kvm -cpu EPYC-v4 -machine q35 -smp 4,maxcpus=255 -m 2048M,slots=5,maxmem=10240M -no-reboot -drive if=pflash,format=raw,unit=0,file=/home/ubuntu/AMDSEV/usr/local/share/qemu/OVMF_CODE.fd,readonly -drive if=pflash,format=raw,unit=1,file=/home/ubuntu/AMDSEV/Image.fd -drive file=/home/ubuntu/AMDSEV/Image.qcow2,if=none,id=disk0,format=qcow2 -device virtio-scsi-pci,id=scsi0,disable-legacy=on,iommu_platform=true -device scsi-hd,drive=disk0 -machine memory-encryption=sev0,vmport=off -object memory-backend-memfd-private,id=ram1,size=2048M,share=true -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,discard=none -machine memory-backend=ram1,kvm-type=protected -nographic -monitor pty -monitor unix:monitor,server,nowait 
Mapping CTRL-C to CTRL-]
Launching VM ...
  /tmp/cmdline.4224
qemu-system-x86_64: -drive if=pflash,format=raw,unit=0,file=/home/ubuntu/AMDSEV/usr/local/share/qemu/OVMF_CODE.fd,readonly: warning: short-form boolean option 'readonly' deprecated
Please use readonly=on instead
char device redirected to /dev/pts/2 (label compat_monitor0)
qemu-system-x86_64: warning: Number of hotpluggable cpus requested (255) exceeds the recommended cpus supported by KVM (192)
qemu-system-x86_64: sev_kvm_init: failed to initialize ret=-5 fw_error=36867 'unknown error'
qemu-system-x86_64: failed to initialize kvm: Operation not permitted

The target machine is an AWS m6a.metal instance with an AMD EPYC 7R13 CPU. We tried the SEV Firmware version 1.55.08 [hex 1.37.07] for EPYC 7xx3 (Milan) from the AMD website, and the older v1.54.01, but the error persists with and without updating the firmware. Catting the three kvm_amd parameters as stated in the readme all return positive ("Y").

Do you have an idea what could cause this issue and how it could be resolved?

Thanks, Zauney

[tlendacky]

Hmm... not sure. Maybe the firmware folks can give some more insight on the 0x9003 error code.

[tlendacky]

Hmm... not sure. Maybe the firmware folks can give some more insight on the 0x9003 error code.

Looks like that is a SPI access error and you may not have the ability to access SPIROM on that machine. Try using the INIT_EX support, which requires the init_ex_path module parameter for the ccp driver.