qemu-system-x86_64: sev_snp_launch_finish: SNP_LAUNCH_FINISH ret=-5 fw_error=11 'Bad measurement'

sza-1 commented 8 months ago

It seems that the PSP measurement check does not work anymore with the latest stable QEMU patches and Direct Linux Boot.

I use the pretty simple command line to launch QEMU and it definitely had worked with the patch series based on QEMU 7.x (and -machine pc-q35-7.2 instead -machine q35):

build/opt/amdese/bin/qemu-system-x86_64 -chardev stdio,id=char0,logfile=logfile,mux=on \
-cpu EPYC-Milan-v2 -device isa-debugcon,chardev=char0,iobase=0x402 \
-drive if=pflash,file=loader,format=raw,readonly=on \
-drive if=pflash,file=nvram,format=raw -enable-kvm \
-kernel kernel \
-m 2G -machine q35,confidential-guest-support=sev0,kvm-type=protected,memory-backend=ram1,vmport=off \
-nic user -no-reboot -nodefaults -nographic \
-object memory-backend-memfd-private,id=ram1,share=true,size=2G \
-object sev-snp-guest,auth-key-enabled=on,cbitpos=51,discard=none,host-data=,id=sev0,id-auth=<IDAUTH>,id-block=<IDBLOCK>,kernel-hashes=on,reduced-phys-bits=1 \
-serial chardev:char0 -smp 1 -trace kvm_sev_*

(There is no initrd param since the kernel is UKI.)

And now I get the following:

kvm_sev_init type SEV-SNP flags 0x0
qemu-system-x86_64: warning: Restricted memory (UPM) enabled, disabling SMM. Memory discard mode: none
kvm_sev_snp_launch_start policy 0x30000 gosvw (null)
kvm_sev_change_state uninit -> launch-update
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 16, size: 80000000, flags: 1
qemu-system-x86_64: warning: creating ROM device with private memory.
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 19, size: 400000, flags: 1
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 21, size: 20000, flags: 0
qemu-system-x86_64: warning: creating ROM device with private memory.
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 23, size: 400000, flags: 1
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 25, size: 20000, flags: 0
kvm_sev_snp_launch_update addr 0x7f18be600000 gpa 0xffc00000 len 0x400000 (Normal page)
kvm_sev_snp_launch_update addr 0x7f1838600000 gpa 0x800000 len 0x9000 (Zero page)
kvm_sev_snp_launch_update addr 0x7f183860a000 gpa 0x80a000 len 0x3000 (Zero page)
kvm_sev_snp_launch_update addr 0x7f183860d000 gpa 0x80d000 len 0x1000 (Secrets page)
kvm_sev_snp_launch_update addr 0x7f183860e000 gpa 0x80e000 len 0x1000 (Cpuid page)
kvm_sev_snp_launch_update addr 0x7f183860f000 gpa 0x80f000 len 0x1000 (Normal page)
kvm_sev_snp_launch_update addr 0x7f1838610000 gpa 0x810000 len 0x10000 (Zero page)
kvm_sev_snp_launch_finish id_block <IDBLOCK> id_auth <IDAUTH> host_data 
qemu-system-x86_64: sev_snp_launch_finish: SNP_LAUNCH_FINISH ret=-5 fw_error=11 'Bad measurement'

LD was calculated with sev-snp-measure, which was not changed last moth in the part of LD calculation algorithm.

Reproduced with the SEV firmware 1.54 and 1.55, EPYC 7B13 (family 25 stepping 1), AMDESE/qemu bbc1bfb6bfb3cde4c22755cedd5b71e651ca35e8, AMDESE/linux 93e2466caee94edb1616ab75f48983ecbc86a03b, AMDESE/ovmf 80318fcdf1bccf5d503197825d62a157efd27c4b.

One more addition: the LD in the attestation report obtained by the guest through /dev/sev-guest (in case when the guest has been booted without IDBlock) differs from the LD calculated by sev-snp-measure.

tlendacky commented 8 months ago

Please be aware that the snp-latest branches need to be used in sync. If you update just one without updating the others you may experience issues. Please update all your components to snp-latest or snp-host-latest as appropriate and see if the issue still persists.

gianlucascopelliti commented 8 months ago

Hi,

I'm also experiencing mismatches of the launch digest compared to the one computed with sev-snp-measure (see https://github.com/virtee/sev-snp-measure/issues/30), after building all components at the same time from the latest commits. I used normal boot (without measuring the kernel).

@sza-1 have you solved your issue?

sza-1 commented 8 months ago

Please be aware that the snp-latest branches need to be used in sync. If you update just one without updating the others you may experience issues. Please update all your components to snp-latest or snp-host-latest as appropriate and see if the issue still persists.

Sure, I always had used snp-latest branch (I mentioned the commits since snp-latest is moving and was pointed to these commits three weeks ago). Thanks.

sza-1 commented 8 months ago

Hi,

I'm also experiencing mismatches of the launch digest compared to the one computed with sev-snp-measure (see virtee/sev-snp-measure#30), after building all components at the same time from the latest commits. I used normal boot (without measuring the kernel).

@sza-1 have you solved your issue?

Not yet. We have currently suspended SEV-SNP support in our product. Maybe, I will try Direct Linux Boot and IDBlock again next month.

larrydewey commented 8 months ago

@sza-1 I have a couple of questions for you:

Are you using the standard OVMF image in your workflow?
Is there anything unique being added to the UKI?
Can you provide us with the command-line arguments you used with sev-snp-measure to calculate your digest?

These will be really helpful in reproducing the error you are seeing.

Thanks, Larry

naoki9911 commented 7 months ago

I have same problem.

Kernel, QEMU, OVMF(OvmfPkg/AmdSev/AmdSevX64.dsc) were built with AMDESE/AMDSEV(https://github.com/AMDESE/AMDSEV/commit/c9413615125c74c04e1ca902165bfdf70086bb70)

Host processor: AMD EPYC 7313P 16-Core Processor
Host kernel: 6.6.0-rc1-snp-host-35f31dfb8101 (https://github.com/AMDESE/linux/commit/35f31dfb81013258126b5391e8c365f9859d203d)
Host distribution: Ubuntu 23.10
Host QEMU: https://github.com/AMDESE/qemu/commit/fe4c9e8e7e7ddac4b19c4366c1f105ffc4a78482
Host OVMF: https://github.com/AMDESE/ovmf/commit/09fbe92dc545779671d7fd89a5bd4f1b14f7e69b
Guest kernel: 6.6.0-snp-guest-690558b32fe6 (https://github.com/AMDESE/linux/commit/690558b32fe6402c309341fbae65f41decb81592)

I updated actual binaries at Google Drive.

I used sev-snp-measure(https://github.com/virtee/sev-snp-measure/commit/015a9eefb5818c0d8f75c28fe0c5410cd23a3369) to calculate hash with following script.

#!/bin/bash

set -eu

SEV_SNP_MEASURE_PATH="/home/naoki/sev-snp-measure"
SEV_SNP_MEASURE="$SEV_SNP_MEASURE_PATH/sev-snp-measure.py"
SEV_SNP_CREATE_IDBLOCK="$SEV_SNP_MEASURE_PATH/snp-create-id-block.py"
QEMU="/home/naoki/AMDSEV/usr/local/bin/qemu-system-x86_64"

CPU_MODEL="EPYC-Milan-v2"
VCPU_NUM="1"

OVMF="OVMF.fd"
KERNEL="vmlinuz-6.6.0-snp-guest-690558b32fe6"
INITRD="initrd.img-6.6.0-snp-guest-690558b32fe6"
APPEND="console=ttyS0 earlyprintk=serial root=/dev/sda2"

ID_KEY="./sev-id.key"
AUTHOR_KEY="./sev-author.key"

MEASURE_HASH=$($SEV_SNP_MEASURE --mode snp --vcpus=$VCPU_NUM --vcpu-type=$CPU_MODEL --ovmf=$OVMF --kernel=$KERNEL --initrd=$INITRD --append="$APPEND" --output-format base64 --vmm-type QEMU)
BLOCKS=$($SEV_SNP_CREATE_IDBLOCK --measure $MEASURE_HASH --idkey $ID_KEY --authorkey $AUTHOR_KEY)
BLOCKS=$(echo $BLOCKS | awk '{print $1}')

set -x

$QEMU -chardev stdio,id=char0,logfile=logfile,mux=on \
-enable-kvm \
-cpu $CPU_MODEL -device isa-debugcon,chardev=char0,iobase=0x402 \
-drive if=pflash,file=$OVMF,format=raw,readonly=on \
-kernel $KERNEL -initrd $INITRD -append "$APPEND" \
-m 2G -machine q35,confidential-guest-support=sev0,kvm-type=protected,memory-backend=ram1,vmport=off \
-nic user -no-reboot -nodefaults -nographic \
-object memory-backend-memfd-private,id=ram1,share=true,size=2G \
-object sev-snp-guest,auth-key-enabled=on,cbitpos=51,discard=none,host-data=,id=sev0,$BLOCKS,kernel-hashes=on,reduced-phys-bits=1 \
-serial chardev:char0 -smp $VCPU_NUM -trace kvm_sev_*

And got following 'Bad measurement' error.

kvm_sev_init type SEV-SNP flags 0x0
qemu-system-x86_64: warning: Restricted memory (UPM) enabled, disabling SMM. Memory discard mode: none
kvm_sev_snp_launch_start policy 0x30000 gosvw (null)
kvm_sev_change_state uninit -> launch-update
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 15, size: 80000000, flags: 1
qemu-system-x86_64: warning: creating ROM device with private memory.
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 18, size: 400000, flags: 1
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 20, size: 20000, flags: 0
qemu-system-x86_64: warning: kvm_create_gmemfd: created memfd: 22, size: 20000, flags: 0
kvm_sev_snp_launch_update addr 0x7f1c21800000 gpa 0xffc00000 len 0x400000 (Normal page)
kvm_sev_snp_launch_update addr 0x7f1b9c600000 gpa 0x800000 len 0x9000 (Zero page)
kvm_sev_snp_launch_update addr 0x7f1b9c60a000 gpa 0x80a000 len 0x3000 (Zero page)
kvm_sev_snp_launch_update addr 0x7f1b9c60d000 gpa 0x80d000 len 0x1000 (Secrets page)
kvm_sev_snp_launch_update addr 0x7f1b9c60e000 gpa 0x80e000 len 0x1000 (Cpuid page)
kvm_sev_snp_launch_update addr 0x7f1b9c60f000 gpa 0x80f000 len 0x1000 (Normal page)
kvm_sev_snp_launch_update addr 0x7f1b9c610000 gpa 0x810000 len 0x10000 (Zero page)
kvm_sev_snp_launch_finish id_block 0MI+/S34o6vF/h0c1PxSCeaMuWiyYX4McnYEGjkJ6WPigPnosXEeovTopMvuNA0FAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAwAAAAAA id_auth AQAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFLou2ht+yHTxSmpZbbs6Yv1PlDaAPGaPHOMHuNFsTslrwRVyql1T+a5Vaaelg5E1AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAALMaTfqLpvEl0+uTcntde0g5WXyMgeEkTbAWIZgObeGtLRQbItY/SH8U1OV5yTgsqwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAANBCCDRZfl00wksO3Tx4o14KU1+JOz2gnRvhft3hSfrCCFIDXYoS11FtA244IHLhegAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOCuQZpIY5RkB2uJ6hQMuD5fHiEHEcneCn7u8GbSO2HuszHHZ4peR65A0516tRUMOQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAuPx5UhbtU59G+IusBM+6uTvUnDLF07XiaBz2Tfho0m1wODS3V9EwpJ5cOoyPIcwcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgWUfcFablggam/IjasPMRYqISy1NOcd16qYj5szhRSWGK5DLhtHgrwEqbjk05arIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAjR4Q3uKn8v1f7BvDwCLzAyhOulXktUJFQ8TiYHCkp9tMZwnGDp+J5NqbhV+ldkk8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACZ7B6M3cYzffikwYsENOWGQ6+0oZYRGZcyvWha1lLztwlOZBGu5BuzLAiL9LilYtEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA== host_data
qemu-system-x86_64: sev_snp_launch_finish: SNP_LAUNCH_FINISH ret=-5 fw_error=11 'Bad measurement'

Below logs are host-side SEV-related dmesg logs.

$ sudo dmesg | grep -i sev
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.6.0-rc1-snp-host-35f31dfb8101 root=/dev/mapper/ubuntu--vg-ubuntu--lv ro mem_encrypt=on kvm_amd.sev=1
[    0.106380] Kernel command line: BOOT_IMAGE=/vmlinuz-6.6.0-rc1-snp-host-35f31dfb8101 root=/dev/mapper/ubuntu--vg-ubuntu--lv ro mem_encrypt=on kvm_amd.se=1
[    0.711481] SEV-SNP: RMP table physical address [0x0000000087800000 - 0x00000000a7dfffff]
[    4.962020] ccp 0000:47:00.1: sev enabled
[    6.996983] ccp 0000:47:00.1: SEV API:1.53 build:5
[    6.996993] ccp 0000:47:00.1: SEV-SNP API:1.53 build:5
[    7.003919] kvm_amd: SEV-ES and SEV-SNP supported: 99 ASIDs
[    7.003920] kvm_amd: SEV enabled (ASIDs 100 - 509)
[    7.003921] kvm_amd: SEV-ES enabled (ASIDs 1 - 99)

mdroth commented 7 months ago

For anyone experiencing this issue, please try the following:

1) stop all guests 2) unload KVM module: rmmod kvm_amd 3) reload KVM module with debug_swap feature disabled: modprobe kvm_amd debug_swap=0 4) retry

Upstream host kernels have a new debug_swap features which modifies the initial state each vCPU's VMSA page, which may affect measurement. Measurement tools will need changes to account for the feature if it is not disabled through the above steps.

gianlucascopelliti commented 7 months ago

For anyone experiencing this issue, please try the following:
1. stop all guests

2. unload KVM module: rmmod kvm_amd

3. reload KVM module with debug_swap feature disabled: modprobe kvm_amd debug_swap=0

4. retry
Upstream host kernels have a new debug_swap features which modifies the initial state each vCPU's VMSA page, which may affect measurement. Measurement tools will need changes to account for the feature if it is not disabled through the above steps.

This works for me. I can confirm that now the measurements match, both in "normal" mode and using linux direct boot. Thanks!

tlendacky commented 7 months ago

Upstream host kernels have a new debug_swap features which modifies the initial state each vCPU's VMSA page, which may affect measurement. Measurement tools will need changes to account for the feature if it is not disabled through the above steps.

Good find @mdroth , the debug_swap feature shows up as a new bit in the SEV_FEATURES within the VMSA. I don't believe that any other part of the VMSA is affected.

Could someone make a local change to the measurement tool to set bit 5 of SEV_FEATURES within the VMSA to see if that resolves the measurement issue when debug_swap is not disabled? (change the init function of vmsa.py to set sev_features = 33 instead of 1)

naoki9911 commented 7 months ago

@mdroth 's approach works for me. Thank you so much!

naoki9911 commented 7 months ago

@tlendacky I tried this and it worked without debug_swap=0 option.

I changed the line (https://github.com/virtee/sev-snp-measure/blob/015a9eefb5818c0d8f75c28fe0c5410cd23a3369/sevsnpmeasure/vmsa.py#L187) to sev_features = 33.

larrydewey commented 7 months ago

@tlendacky @mdroth I verified the change, as well, and opened the PR mentioned above to begin introducing the patch to fix this.

tlendacky commented 7 months ago

@tlendacky @mdroth I verified the change, as well, and opened the PR mentioned above to begin introducing the patch to fix this.

There should probably be command line parameters that can be used to apply/set bits in SEV_FEATURES as needed.

sza-1 commented 7 months ago

@sza-1 I have a couple of questions for you:
1. Are you using the standard OVMF image in your workflow?

2. Is there anything unique being added to the UKI?

3. Can you provide us with the command-line arguments you used with `sev-snp-measure` to calculate your digest?
These will be really helpful in reproducing the error you are seeing.

Thanks, Larry

Yes. I built it from the snp-latest.
No and yes. I use systemd's ukify.py to combine the pretty default initramfs and AMDESE's kernel into PE executable. But I use a custom initramfs-tools profile, which should not (and it had not affected until last times) affect SEV-SNP. E.g.: ukify --cmdline="${CMDLINE} boot=test" /boot/vmlinuz-* /boot/initrd.img-*.
sev-snp-measure --kernel *.efi --mode=snp --ovmf=OVMF.fd --vcpu-sig=0x800f12 --vcpus=1 > ld-800f12-1.

You may see here the full Dockerfile I wrote for this purpose, if you wish: https://github.com/sza-1/sev/blob/main/Dockerfile.

larrydewey commented 7 months ago

@tlendacky @mdroth I verified the change, as well, and opened the PR mentioned above to begin introducing the patch to fix this.

There should probably be command line parameters that can be used to apply/set bits in SEV_FEATURES as needed.

I completely agree. @mdroth and I were talking about that yesterday. Diego and I will adjust the existing PR to incorporate those changes.

larrydewey commented 7 months ago

@sza-1 I have a couple of questions for you:
1. Are you using the standard OVMF image in your workflow?

2. Is there anything unique being added to the UKI?

3. Can you provide us with the command-line arguments you used with `sev-snp-measure` to calculate your digest?
These will be really helpful in reproducing the error you are seeing. Thanks, Larry
Yes. I built it from the snp-latest.

No and yes. I use systemd's ukify.py to combine the pretty default initramfs and AMDESE's kernel into PE executable. But I use a custom initramfs-tools profile, which should not (and it had not affected until last times) affect SEV-SNP. E.g.: ukify --cmdline="${CMDLINE} boot=test" /boot/vmlinuz-* /boot/initrd.img-*.

sev-snp-measure --kernel *.efi --mode=snp --ovmf=OVMF.fd --vcpu-sig=0x800f12 --vcpus=1 > ld-800f12-1.

You may see here the full Dockerfile I wrote for this purpose, if you wish: https://github.com/sza-1/sev/blob/main/Dockerfile.

Awesome, thanks for the info! I will make sure we add this to our list of test-cases to add to our internal CI.

AMDESE / AMDSEV

qemu-system-x86_64: sev_snp_launch_finish: SNP_LAUNCH_FINISH ret=-5 fw_error=11 'Bad measurement' #195