[BUG] snp guest vm failed to boot with large memory

[JustPlay]

I use the following qemu cmd to launch a snp guest /usr/bin/qemu-system-x86_64 -enable-kvm -no-reboot -nographic -monitor pty -qmp tcp::5555,server,nowait -cpu EPYC-v4 -machine q35 -smp 16,maxcpus=16 -m 131072M,slots=11,maxmem=133120M -drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=/root/sev/images/ubuntu/jammy.OVMF_VARS -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic0,romfile= -netdev user,id=vmnic0,hostfwd=tcp::2222-:22 -drive if=virtio,id=disk0,format=qcow2,file=/root/sev/images/ubuntu/jammy.qcow2 -machine memory-encryption=sev0,vmport=off -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1

the snp vm failed to boot launch.log_128G_fail.txt

launch.log ... [ 12.911854] fbcon: bochs-drmdrmfb (fb0) is primary device [ 12.958204] Console: switching to colour frame buffer device 128x48 [ 12.961793] bochs-drm 0000:00:01.0: [drm] fb0: bochs-drmdrmfb frame buffer device // it hung at here for some time ... Scanning for Btrfs filesystems // it hung at here for some time again, the fail ...

16GB fail too.

However, when i use 8GB memory, it works well (all other qemu args are same) /usr/bin/qemu-system-x86_64 -enable-kvm -no-reboot -nographic -monitor pty -qmp tcp::5555,server,nowait -cpu EPYC-v4 -machine q35 -smp 16,maxcpus=16 -m 8192M,slots=11,maxmem=10240M -drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=/root/sev/images/ubuntu/jammy.OVMF_VARS -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic0,romfile= -netdev user,id=vmnic0,hostfwd=tcp::2222-:22 -drive if=virtio,id=disk0,format=qcow2,file=/root/sev/images/ubuntu/jammy.qcow2 -machine memory-encryption=sev0,vmport=off -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 launch.log_8GB_succ.txt

and, sev-es works weil with 128GB launch.log_128G_succ_sev_es.txt

host: host os: debian-10 qemu: https://github.com/AMDESE/qemu.git,snp-v3 ovmf: https://github.com/tianocore/edk2.git,edk2-stable202208 host kernel: https://github.com/AMDESE/linux.git,sev-snp-iommu-avic_5.19-rc6_v3 firmware: amd_sev_fam19h_model0xh_1.51.03

guest: guest os: ubuntu-22.04 guest kernel: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git,v6.0

[JustPlay]

numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 node 0 size: 1019699 MB node 0 free: 9062 MB node 1 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node 1 size: 1023882 MB node 1 free: 198221 MB node distances: node 0 1 0: 10 32 1: 32 10

lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 43 bits physical, 48 bits virtual CPU(s): 256 On-line CPU(s) list: 0-255 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 2 NUMA node(s): 2 Vendor ID: AuthenticAMD CPU family: 25 Model: 1 Model name: AMD EPYC 7Y83 64-Core Processor Stepping: 1 CPU MHz: 2607.761 CPU max MHz: 3854.4431 CPU min MHz: 1500.0000 BogoMIPS: 4890.82 Virtualization: AMD-V L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 32768K NUMA node0 CPU(s): 0-63,128-191 NUMA node1 CPU(s): 64-127,192-255 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin brs arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca sme sev sev_es sev_snp

[JustPlay]

after i disabled btrfs logical, an other error occured:

Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. [ 364.077461] INFO: task systemd-udevd:257 blocked for more than 120 seconds. [ 364.078362] Not tainted 6.0.0-sev-snp-guest-v6.0 #1 [ 364.078920] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 364.079707] task:systemd-udevd state:D stack: 0 pid: 257 ppid: 246 flags:0x00004006 [ 364.080537] Call Trace: [ 364.080797] [ 364.081025] schedule+0x31b/0x1470 [ 364.081427] ? filemap_invalidate_unlock_two+0x50/0x50 [ 364.081967] schedule+0x6b/0x110 [ 364.082305] io_schedule+0x46/0x80 [ 364.082665] folio_wait_bit_common+0x140/0x390 [ 364.083119] ? filemap_invalidate_unlock_two+0x50/0x50 [ 364.083648] ? blkdev_writepages+0x20/0x20 [ 364.084341] do_read_cache_folio+0x11b/0x190 [ 364.085008] ? osf_partition+0x180/0x180 [ 364.085654] read_cache_folio+0x12/0x20 [ 364.086266] read_part_sector+0x39/0xf0 [ 364.086883] sgi_partition+0x33/0x190 [ 364.087480] ? snprintf+0x49/0x70 [ 364.088036] ? osf_partition+0x180/0x180 [ 364.088656] bdev_disk_changed+0x1e8/0x480 [ 364.089286] blkdev_get_whole+0x81/0xa0 [ 364.089923] blkdev_get_by_dev.part.0+0x142/0x320 [ 364.090625] blkdev_get_by_dev+0x55/0x70 [ 364.091243] disk_scan_partitions+0x4b/0x80 [ 364.091887] device_add_disk+0x3be/0x3d0 [ 364.092506] virtblk_probe+0x49d/0x7f0 [virtio_blk] [ 364.093212] virtio_dev_probe+0x1b6/0x2b0 [ 364.093852] really_probe+0xff/0x3b0 [ 364.094423] ? pm_runtime_barrier+0x55/0x90 [ 364.095057] driver_probe_device+0x8a/0x190 [ 364.095707] driver_probe_device+0x23/0xc0 [ 364.096317] driver_attach+0xbb/0x230 [ 364.096913] ? device_attach_driver+0x140/0x140 [ 364.097613] bus_for_each_dev+0x7f/0xd0 [ 364.098198] driver_attach+0x1e/0x30 [ 364.098757] bus_add_driver+0x1c5/0x220 [ 364.099335] driver_register+0x95/0x100 [ 364.099913] register_virtio_driver+0x20/0x40 [ 364.100541] virtio_blk_init+0x56/0x1000 [virtio_blk] [ 364.101228] ? 0xffffffffc053a000 [ 364.101775] do_one_initcall+0x49/0x210 [ 364.102348] ? kmem_cache_alloc_trace+0x160/0x300 [ 364.103011] do_init_module+0x52/0x210 [ 364.103579] load_module+0x1f54/0x2380 [ 364.104143] ? security_kernel_post_read_file+0x5c/0x70 [ 364.104852] do_sys_finit_module+0xc8/0x140 [ 364.105498] ? do_sys_finit_module+0xc8/0x140 [ 364.106145] x64_sys_finit_module+0x18/0x20 [ 364.106764] do_syscall_64+0x5c/0x90 [ 364.107296] ? syscall_exit_to_user_mode+0x26/0x50 [ 364.107947] ? x64_sys_newfstatat+0x1c/0x30 [ 364.108558] ? do_syscall_64+0x69/0x90 [ 364.109102] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 364.109792] RIP: 0033:0x7f2d945c3a3d [ 364.110321] RSP: 002b:00007ffcc5b6b398 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 364.111236] RAX: ffffffffffffffda RBX: 0000561215ab8eb0 RCX: 00007f2d945c3a3d [ 364.112117] RDX: 0000000000000000 RSI: 00007f2d9475a441 RDI: 0000000000000005 [ 364.113002] RBP: 0000000000020000 R08: 0000000000000000 R09: 00007ffcc5b6b4d0 [ 364.113905] R10: 0000000000000005 R11: 0000000000000246 R12: 00007f2d9475a441 [ 364.114795] R13: 0000561215ab2f90 R14: 0000561215ab5ba0 R15: 0000561215ab7210 [ 364.115678] [ 364.116097] INFO: task blkid:418 blocked for more than 120 seconds. [ 364.116901] Not tainted 6.0.0-sev-snp-guest-v6.0 #1 [ 364.117639] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 364.118595] task:blkid state:D stack: 0 pid: 418 ppid: 417 flags:0x00000002 [ 364.119605] Call Trace: [ 364.120051] [ 364.120472] schedule+0x31b/0x1470 [ 364.121028] ? _raw_spin_unlock+0x19/0x40 [ 364.121650] schedule+0x6b/0x110 [ 364.122176] schedule_preempt_disabled+0x18/0x30 [ 364.122835] mutex_lock.constprop.0+0x3ef/0x7a0 [ 364.123505] ? ilookup+0xa9/0x100 [ 364.124034] __mutex_lock_slowpath+0x13/0x20 [ 364.124663] mutex_lock+0x38/0x50 [ 364.125195] blkdev_get_by_dev.part.0+0x107/0x320 [ 364.125882] blkdev_get_by_dev+0x55/0x70 [ 364.126479] ? blkdev_close+0x30/0x30 [ 364.127042] blkdev_open+0x50/0xa0 [ 364.127591] do_dentry_open+0x15a/0x3f0 [ 364.128170] vfs_open+0x2d/0x40 [ 364.128687] path_openat+0xac7/0x10d0 [ 364.129246] ? mntput+0x24/0x40 [ 364.129784] do_filp_open+0xb2/0x160 [ 364.130336] ? _raw_spin_unlock+0x19/0x40 [ 364.130938] ? alloc_fd+0xb9/0x190 [ 364.131480] do_sys_openat2+0x9f/0x160 [ 364.132047] x64_sys_openat+0x55/0x90 [ 364.132632] do_syscall_64+0x5c/0x90 [ 364.133181] ? syscall_exit_to_user_mode+0x26/0x50 [ 364.133865] ? x64_sys_close+0x11/0x50 [ 364.134447] ? do_syscall_64+0x69/0x90 [ 364.135022] ? syscall_exit_to_user_mode+0x26/0x50 [ 364.135689] ? do_syscall_64+0x69/0x90 [ 364.136253] ? debug_smp_processor_id+0x17/0x20 [ 364.136905] ? fpregs_assert_state_consistent+0x2a/0x50 [ 364.137634] ? exit_to_user_mode_prepare+0x49/0x1a0 [ 364.138309] ? irqentry_exit_to_user_mode+0x9/0x20 [ 364.138981] ? irqentry_exit+0x3b/0x50 [ 364.139550] ? exc_page_fault+0x87/0x180 [ 364.140128] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 364.140822] RIP: 0033:0x7f9c7370c6eb [ 364.141382] RSP: 002b:00007ffd24b254a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 364.142314] RAX: ffffffffffffffda RBX: 00005617296235a0 RCX: 00007f9c7370c6eb [ 364.143206] RDX: 0000000000080800 RSI: 0000561729634870 RDI: 00000000ffffff9c [ 364.144100] RBP: 0000561729634870 R08: 0000000000000000 R09: 0000561729634910 [ 364.144998] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000080800 [ 364.145906] R13: 8000000000000000 R14: 800000006343c40f R15: 0000000000000000 [ 364.146797]

[JustPlay]

i think it related to swiotlb

[tlendacky]

i think it related to swiotlb

It appears so, try adding "swiotlb=131072" to the kernel command line to limit the size of SWIOTLB.

[JustPlay]

why limit swiotlb size?

@tlendacky

i thiink larger swiotlb size is good for performance i have tried to enlarge swiotlb to 1GB, but if i do so, even guest memory size is 8GB, snp vm will NOT boot (if use default swiotlb size (512M), 8GB snp vm can boot)

limit swiotlb to 512MB can boot all vm

i still want to know why swiotlb size impact sev|es|snp vm booting? thanks

[tlendacky]

Because if you look at your logs the SWIOTLB allocation is failing because it can't allocate 1GB of contiguous memory. That used to cause a panic and make it easy to identify the issue, but because of some issues with other architectures, you now you just get a warning.

[JustPlay]

Because if you look at your logs the SWIOTLB allocation is failing because it can't allocate 1GB of contiguous memory. That used to cause a panic and make it easy to identify the issue, but because of some issues with other architectures, you now you just get a warning.

i found that sev|es encrypted boot can use 1GB swiotlb https://blog.hansenpartnership.com/deploying-encrypted-images-for-confidential-computing/

my testing data: guest memory size : swiotlb size in es-encrypted boot vm | swiotlb size in normal es vm 8G:491M|512M, 12G:773M|FAIL, 16G:982M|FAIL, 32G:1024M|FAIL, 64G:1024M|FAIL, 128G:1024M|FAIL swiotlb size taken from dmesg

i see the kernel code

   /*
     * By default allocate the bounce buffer memory from low memory, but
     * allow to pick a location everywhere for hypervisors with guest
     * memory encryption.
     */
retry:
    bytes = PAGE_ALIGN(nslabs << IO_TLB_SHIFT);
    if (flags & SWIOTLB_ANY)
        tlb = memblock_alloc(bytes, PAGE_SIZE);
    else
        tlb = memblock_alloc_low(bytes, PAGE_SIZE);
    if (!tlb) {
        pr_warn("%s: failed to allocate tlb structure\n", __func__);
        return;
    }

so, in encrypted boot, it seems the memory was allocated from non-low memory, so can we use non-low memory for non-encrypted boot?

update: it use memblock_alloc_low() in both encrypted-boot and normal boot

[tlendacky]

update: it use memblock_alloc_low() in both encrypted-boot and normal boot

Correct, SWIOTLB by default will use low memory. If there is not enough contiguous memory below 4GB, the allocation fails, and then the boot of an SEV guest will likely fail.

I have been able to boot with 1GB of SWIOTLB, but it can be hit or miss and may depend on the devices present, number of vCPUs, etc.