search

NVIDIA / nvtrust

Ancillary open source software to support confidential computing on NVIDIA GPUs
Apache License 2.0
175 stars 27 forks source link

NVidia CC with Intel TDX #54

Open YurkoWasHere opened 1 month ago

YurkoWasHere commented 1 month ago

Followed the instructions from scratch from the nvidia pdf (as a reference implementation starting point)

After boot i can see the device with nvidia-smi but then i see the following in dmesg

[   83.133585] ACPI Warning: \_SB.PCI0.S18.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20221020/nsarguments-61)
[   86.776417] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[   86.776425] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[   87.352832] NVRM: nvCheckOkFailedNoLog: Check failed: Call not supported [NV_ERR_NOT_SUPPORTED] (0x00000056) returned from pEntry->pCallback(pGpu, pEntry->pCallbackParam) @ kernel_fifo.c:3021
[   88.578481] NVRM: kgspCheckGspRmCcCleanup_GH100: CC secret cleanup successful!

after that i cannot access nvidia-smi again and instead yeilds:

[  121.802181] NVRM: kfspPollForResponse_IMPL: FSP command timed out
[  121.802187] NVRM: kfspSendBootCommands_GH100: Sent following content to FSP: 
[  121.802189] NVRM: kfspSendBootCommands_GH100: version=0x1, size=0x35c, gspFmcSysmemOffset=0x12b040000
[  121.802190] NVRM: kfspSendBootCommands_GH100: frtsSysmemOffset=0x0, frtsSysmemSize=0x0
[  121.802191] NVRM: kfspSendBootCommands_GH100: frtsVidmemOffset=0x200000, frtsVidmemSize=0x100000
[  121.802192] NVRM: kfspSendBootCommands_GH100: gspBootArgsSysmemOffset=0x112f1f000
[  121.802193] NVRM: kfspSendBootCommands_GH100: FSP boot cmds failed. RM cannot boot.
[  121.802195] NVRM: kfspDumpDebugState_GH100: FSP microcode v4.76
[  121.802196] NVRM: kfspDumpDebugState_GH100: GPU 0000:01:00
[  121.802197] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(0) = 0x9f
[  121.802199] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(1) = 0x110186
[  121.802201] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(2) = 0x1103c0
[  121.802202] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(3) = 0x5
[  121.803188] NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kfspSendBootCommands_HAL(pGpu, pKernelFsp) @ kernel_gsp_gh100.c:756
[  121.803763] NVRM: gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110804,  regvalue: 0xbadf4100,  error code: Unknown SYS_PRI_ERROR_CODE
[  121.803767] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[  122.883280] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x65:1784)
[  122.885149] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
benschlueter commented 1 month ago

I have a similar problem (550.78)

[ 1223.160374] ACPI Warning: \_SB.PCI0.S20.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[ 1225.896461] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[ 1225.896471] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[ 1225.917683] NVRM: nvAssertOkFailedNoLog: Assertion failed: Invalid data passed [NV_ERR_INVALID_DATA] (0x00000025) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_INTERNAL_MEMMGR_MEMORY_TRANSFER_WITH_GSP, &gspParams, sizeof(gspParams)) @ mem_utils.c:283
[ 1225.917814] NVRM: nvAssertOkFailedNoLog: Assertion failed: Invalid data passed [NV_ERR_INVALID_DATA] (0x00000025) returned from _memmgrMemReadOrWriteWithGsp(pGpu, pDstInfo, pBuf, size, NV_FALSE ) @ mem_utils.c:864
[ 1225.917821] NVRM: nvAssertFailedNoLog: Assertion failed: status == NV_OK @ mem_mgr.c:607
[ 1225.917836] NVRM: nvAssertOkFailedNoLog: Assertion failed: Invalid data passed [NV_ERR_INVALID_DATA] (0x00000025) returned from memmgrVerifyGspDmaOps(pGpu, GPU_GET_MEMORY_MANAGER(pGpu)) @ kern_bus_gm107.c:385
[ 1225.917843] NVRM: RmInitNvDevice: *** Cannot initialize the device
[ 1225.917844] NVRM: RmInitAdapter: RmInitNvDevice failed, bailing out of RmInitAdapter
[ 1225.917865] NVRM: rmapiReportLeakedDevices: Device object leak: (0xc1e00003, 0xcaf00000). Please file a bug against RM-core.
[ 1225.917866] NVRM: nvAssertFailedNoLog: Assertion failed: 0 @ rmapi.c:961
[ 1226.291562] NVOC: __nvoc_objDelete: Child class PrereqTracker not freed from parent class OBJGPU.NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x24:0x25:1025)
[ 1226.293656] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 1226.375951] nvidia-uvm: Loaded the UVM driver, major device number 244.
YurkoWasHere commented 1 month ago

Has any one gotten H100 and Intel TDX working together?

YurkoWasHere commented 1 month ago

My guess is it stems from this

[  159.931558] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[  159.931565] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[  160.490229] NVRM: nvCheckOkFailedNoLog: Check failed: Call not supported [NV_ERR_NOT_SUPPORTED] (0x00000056) returned from pEntry->pCallback(pGpu, pEntry->pCallbackParam) @ kernel_fifo.c:3021
[  161.794999] NVRM: kgspCheckGspRmCcCleanup_GH100: CC secret cleanup successful!

I think As CC is wiped the H100 becomes uninitialized in the CC environment.?

https://github.com/NVIDIA/open-gpu-kernel-modules/blob/550/src/nvidia/generated/g_kern_gmmu_nvoc.h#L1967

YurkoWasHere commented 1 month ago

I rebuild the host using the newer https://github.com/canonical/tdx repo

I'm left with a similar issue as soon as i run nvidia-smi (or nvidia-persisted starts)

[    9.515073] nvidia: loading out-of-tree module taints kernel.
[    9.515084] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    9.574563] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
[    9.574570] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  550.54.15  Release Build  (root@tdx-guest)  Fri May  3 20:14:30 UTC 2024
[    9.597528] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  550.54.15  Release Build  (root@tdx-guest)  Fri May  3 20:15:01 UTC 2024
[    9.646394] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[    9.646398] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 0
...
  59.452702] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[   59.452710] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[   59.482016] Encryption FAILED
[   59.482020] NVRM: _memmgrMemReadOrWriteWithGsp: Fatal error detected in GSP-DMA decrypt: 0x25!
[   59.482022] NVRM: confComputeSetErrorState_KERNEL: ConfCompute: Fatal error hit!
[   59.483359] NVRM: nvAssertOkFailedNoLog: Assertion failed: Invalid data passed [NV_ERR_INVALID_DATA] (0x00000025) returned from _memmgrMemReadOrWriteWithGsp(pGpu, pSrcInfo, pBuf, size, NV_TRUE ) @ mem_utils.c:935
[   59.483425] NVRM: nvAssertFailedNoLog: Assertion failed: status == NV_OK @ mem_mgr.c:612
[   59.483432] NVRM: nvAssertOkFailedNoLog: Assertion failed: Invalid data passed [NV_ERR_INVALID_DATA] (0x00000025) returned from memmgrVerifyGspDmaOps(pGpu, GPU_GET_MEMORY_MANAGER(pGpu)) @ kern_bus_gm107.c:385
[   59.483437] NVRM: RmInitNvDevice: *** Cannot initialize the device
[   59.483438] NVRM: RmInitAdapter: RmInitNvDevice failed, bailing out of RmInitAdapter
[   59.483450] NVRM: rmapiReportLeakedDevices: Device object leak: (0xc1e00005, 0xcaf00000). Please file a bug against RM-core.
[   59.483452] NVRM: nvAssertFailedNoLog: Assertion failed: 0 @ rmapi.c:961
[   59.483466] NVRM: GspMsgQueueSendCommand: Encryption failed with status = 0x1f.
[   59.483467] NVRM: _kgspRpcSendMessage: GspMsgQueueSendCommand failed on GPU0: 0x1f
[   59.483469] NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000001f for fn 10!
[   59.483471] NVRM: rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1e00005; hObject=0xcaf00001; paramsStatus=0x00000000; status=0x0000001f
[   59.483476] NVRM: GspMsgQueueSendCommand: Encryption failed with status = 0x1f.
[   59.483478] NVRM: _kgspRpcSendMessage: GspMsgQueueSendCommand failed on GPU0: 0x1f
[   59.483479] NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000001f for fn 10!
[   59.483480] NVRM: rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1e00005; hObject=0xcaf00000; paramsStatus=0x00000000; status=0x0000001f
[   60.341833] NVOC: __nvoc_objDelete: Child class PrereqTracker not freed from parent class OBJGPU.NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x24:0x25:1025)
[   60.344060] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[   65.675508] NVRM: kfspPollForResponse_IMPL: FSP command timed out
[   65.675514] NVRM: kfspSendBootCommands_GH100: Sent following content to FSP:
[   65.675516] NVRM: kfspSendBootCommands_GH100: version=0x1, size=0x35c, gspFmcSysmemOffset=0x1ccc0000
[   65.675517] NVRM: kfspSendBootCommands_GH100: frtsSysmemOffset=0x0, frtsSysmemSize=0x0
[   65.675519] NVRM: kfspSendBootCommands_GH100: frtsVidmemOffset=0x200000, frtsVidmemSize=0x100000
[   65.675520] NVRM: kfspSendBootCommands_GH100: gspBootArgsSysmemOffset=0x205b8000
[   65.675521] NVRM: kfspSendBootCommands_GH100: FSP boot cmds failed. RM cannot boot.
[   65.675523] NVRM: kfspDumpDebugState_GH100: FSP microcode v4.76
[   65.675524] NVRM: kfspDumpDebugState_GH100: GPU 0000:01:00
[   65.675526] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(0) = 0x9f
[   65.675528] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(1) = 0x110186
[   65.675530] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(2) = 0x1103c0
[   65.675531] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(3) = 0x5
[   65.676474] NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kfspSendBootCommands_HAL(pGpu, pKernelFsp) @ kernel_gsp_gh100.c:756
[   65.677035] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[   66.625260] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x65:1784)
[   66.627423] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[   71.975694] NVRM: kfspPollForResponse_IMPL: FSP command timed out
[   71.975699] NVRM: kfspSendBootCommands_GH100: Sent following content to FSP:
[   71.975701] NVRM: kfspSendBootCommands_GH100: version=0x1, size=0x35c, gspFmcSysmemOffset=0x1cd00000
[   71.975702] NVRM: kfspSendBootCommands_GH100: frtsSysmemOffset=0x0, frtsSysmemSize=0x0
[   71.975703] NVRM: kfspSendBootCommands_GH100: frtsVidmemOffset=0x200000, frtsVidmemSize=0x100000
[   71.975704] NVRM: kfspSendBootCommands_GH100: gspBootArgsSysmemOffset=0x74e97000
[   71.975705] NVRM: kfspSendBootCommands_GH100: FSP boot cmds failed. RM cannot boot.
[   71.975708] NVRM: kfspDumpDebugState_GH100: FSP microcode v4.76
[   71.975709] NVRM: kfspDumpDebugState_GH100: GPU 0000:01:00
[   71.975711] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(0) = 0x9f
[   71.975713] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(1) = 0x110186
[   71.975714] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(2) = 0x1103c0
[   71.975716] NVRM: kfspDumpDebugState_GH100: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(3) = 0x5
[   71.976588] NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kfspSendBootCommands_HAL(pGpu, pKernelFsp) @ kernel_gsp_gh100.c:756
[   71.976902] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[   73.123861] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x65:1784)
[   73.125978] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[   73.267261] nvidia-uvm: Loaded the UVM driver, major device number 509.

lsmod

lsmod
Module                  Size  Used by
cfg80211             1306624  0
binfmt_misc            24576  1
nls_iso8859_1          12288  1
intel_rapl_msr         20480  0
intel_rapl_common      36864  1 intel_rapl_msr
tdx_guest              12288  0
intel_uncore_frequency_common    16384  0
nfit                   81920  0
nvidia_drm            122880  0
nvidia_modeset       1495040  1 nvidia_drm
nvidia               8667136  1 nvidia_modeset
ecdh_generic           16384  0
ecdsa_generic          16384  0
drm_kms_helper        270336  1 nvidia_drm
vmw_vsock_virtio_transport    20480  0
video                  73728  1 nvidia_modeset
i2c_i801               36864  0
vmw_vsock_virtio_transport_common    57344  1 vmw_vsock_virtio_transport
i2c_smbus              16384  1 i2c_i801
wmi                    40960  1 video
ecc                    45056  3 ecdh_generic,ecdsa_generic,nvidia
vsock                  61440  2 vmw_vsock_virtio_transport_common,vmw_vsock_virtio_transport
lpc_ich                32768  0
input_leds             12288  0
mac_hid                12288  0
serio_raw              20480  0
dm_multipath           45056  0
msr                    12288  0
drm                   765952  4 drm_kms_helper,nvidia,nvidia_drm
efi_pstore             12288  0
dmi_sysfs              20480  0
qemu_fw_cfg            24576  0
ip_tables              36864  0
x_tables               69632  1 ip_tables
autofs4                57344  2
btrfs                1990656  0
blake2b_generic        24576  0
raid10                 77824  0
raid456               200704  0
async_raid6_recov      20480  1 raid456
async_memcpy           16384  2 raid456,async_raid6_recov
async_pq               20480  2 raid456,async_raid6_recov
async_xor              16384  3 async_pq,raid456,async_raid6_recov
async_tx               16384  5 async_pq,async_memcpy,async_xor,raid456,async_raid6_recov
xor                    20480  2 async_xor,btrfs
raid6_pq              126976  4 async_pq,btrfs,raid456,async_raid6_recov
libcrc32c              12288  2 btrfs,raid456
raid1                  57344  0
raid0                  24576  0
multipath              20480  0
linear                 16384  0
crct10dif_pclmul       12288  1
crc32_pclmul           12288  0
polyval_clmulni        12288  0
polyval_generic        12288  1 polyval_clmulni
ghash_clmulni_intel    16384  0
aesni_intel           356352  0
crypto_simd            16384  1 aesni_intel
ahci                   49152  0
psmouse               212992  0
cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
libahci                57344  1 ahci
Tan-YiFan commented 1 month ago

Please make sure that nvidia-persistenced is executed before any GPU-related commands (including nvidia-smi).

YurkoWasHere commented 1 month ago

I always run nvidia-persistenced right after i insmod the drivers.

I managed to get it to run on the old kernel, but when i go with a minimalist OS it crashes.

Nvidia drivers installed

[  316.068999] ACPI: bus type drm_connector registered
[  316.072183] cryptd: max_cpu_qlen set to 1000
[  316.078102] AVX2 version of gcm_enc/dec engaged.
[  316.079343] AES CTR mode by8 optimization enabled
[  316.092021] nvidia: loading out-of-tree module taints kernel.
[  316.177599] nvidia-nvlink: Nvlink Core is being initialized, major device number 236
[  316.177831] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  550.54.15  Release Build  (root@ubuntu)  Wed May  8 12:12:31 UTC 2024
[  316.276342] nvidia-uvm: Loaded the UVM driver, major device number 234.

About 30 seconds later while the system was configufing the network (ifconfig eth0 up.. dhcpd. etc)


[  350.486784] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [insmod:2277]
[  350.487023] Modules linked in: nvidia_uvm(OE) nvidia(OE) aesni_intel(E) crypto_simd(E) cryptd(E) drm(E) ecdsa_generic(E) ecdh_generic(E) ecc(E) e1000(E) virtio_scsi(E) vmw_vsock_virtio_transport(E) vhost_vsock(E) vmw_vsock_virtio_transport_common(E) vsock(E) vhost(E) vhost_iotlb(E) dm_integrity(E) dm_bufio(E) async_xor(E) async_tx(E) xor(E) essiv(E) authenc(E) dm_crypt(E)
[  350.487717] CPU: 1 PID: 2277 Comm: insmod Tainted: G           OE       6.2.0-mvp10v1+8-generic #mvp10v1+tdx
[  350.487990] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown unknown
[  350.488161] RIP: 0010:smp_call_function_many_cond+0x11e/0x3b0
[  350.488315] Code: 76 35 4c 63 e8 4c 8b 23 49 81 fd ff 1f 00 00 0f 87 4e 02 00 00 4a 8b 34 ed 80 7a e7 8b 4c 01 e6 8b 56 08 83 e2 01 74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb b1 48 83 c4 40 5b 41 5c 41 5d
[  350.488722] RSP: 0018:ff61389300cb7ae0 EFLAGS: 00000202
[  350.488845] RAX: 0000000000000006 RBX: ff4d95ab7fa72780 RCX: 0000000000000001
[  350.489009] RDX: 0000000000000001 RSI: ff4d95ab7fbb88a0 RDI: ff4d959db9466a88
[  350.489265] RBP: ff61389300cb7b48 R08: 0000000000000006 R09: 0000000000000000
[  350.489436] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000388a0
[  350.489600] R13: 0000000000000006 R14: 0000000000000001 R15: ff4d95ab7fa72780
[  350.489764] FS:  00007f8fea453000(0000) GS:ff4d95ab7fa40000(0000) knlGS:0000000000000000
[  350.489937] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  350.490079] CR2: 0000564dea5f05f0 CR3: 000000070c4aa006 CR4: 0000000000771ee0
[  350.490238] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  350.490407] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  350.490580] PKRU: 55555554
[  350.490655] Call Trace:
[  350.490719]  <TASK>
[  350.490773]  ? __pfx_do_flush_tlb_all+0x10/0x10
[  350.490884]  on_each_cpu_cond_mask+0x24/0x40
[  350.490993]  flush_tlb_kernel_range+0x41/0xc0
[  350.491109]  __purge_vmap_area_lazy+0xcf/0x780
[  350.491219]  ? purge_fragmented_blocks_allcpus+0x51/0x210
[  350.491337]  _vm_unmap_aliases+0x129/0x180
[  350.491421]  __vunmap+0x1a0/0x320
[  350.491503]  __vfree+0x22/0x70
[  350.491593]  vfree+0x2c/0x50
[  350.491677]  module_memfree+0x1c/0x30
[  350.491759]  load_module+0x763/0xc90
[  350.491876]  ? security_kernel_post_read_file+0x5c/0x70
[  350.491988]  ? kernel_read_file+0x24e/0x2a0
[  350.492086]  __do_sys_finit_module+0xc8/0x140
[  350.492195]  ? __do_sys_finit_module+0xc8/0x140
[  350.492304]  __x64_sys_finit_module+0x18/0x20
[  350.492414]  do_syscall_64+0x59/0x90
[  350.492499]  ? exit_to_user_mode_prepare+0x37/0xb0
[  350.492607]  ? syscall_exit_to_user_mode+0x26/0x50
[  350.492725]  ? __x64_sys_mmap+0x33/0x50
[  350.492795]  ? do_syscall_64+0x69/0x90
[  350.492877]  ? exit_to_user_mode_prepare+0x37/0xb0
[  350.492996]  ? irqentry_exit_to_user_mode+0x9/0x20
[  350.493104]  ? irqentry_exit+0x3b/0x50
[  350.493187]  ? exc_page_fault+0x87/0x180
[  350.493268]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  350.493382] RIP: 0033:0x7f8fe9d1e88d
[  350.493466] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 73 b5 0f 00 f7 d8 64 89 01 48
[  350.493858] RSP: 002b:00007fff79091e48 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[  350.494023] RAX: ffffffffffffffda RBX: 0000564deb44f490 RCX: 00007f8fe9d1e88d
[  350.494185] RDX: 0000000000000000 RSI: 0000564dea5f8cd2 RDI: 0000000000000003
[  350.494350] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  350.494518] R10: 0000000000000003 R11: 0000000000000246 R12: 0000564dea5f8cd2
[  350.494683] R13: 0000564deb44e3d0 R14: 0000564dea5f7888 R15: 0000564deb44f5a0
[  350.494898]  </TASK>
[  378.318793] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  378.319013] rcu:     6-...0: (0 ticks this GP) idle=5b1c/1/0x4000000000000000 softirq=2306/2306 fqs=6376
[  378.319194]  (detected by 12, t=15002 jiffies, g=7201, q=944 ncpus=16)
[  378.319334] Sending NMI from CPU 12 to CPUs 6:
[  378.486784] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [insmod:2277]
[  378.486920] Modules linked in: nvidia_uvm(OE) nvidia(OE) aesni_intel(E) crypto_simd(E) cryptd(E) drm(E) ecdsa_generic(E) ecdh_generic(E) ecc(E) e1000(E) virtio_scsi(E) vmw_vsock_virtio_transport(E) vhost_vsock(E) vmw_vsock_virtio_transport_common(E) vsock(E) vhost(E) vhost_iotlb(E) dm_integrity(E) dm_bufio(E) async_xor(E) async_tx(E) xor(E) essiv(E) authenc(E) dm_crypt(E)
[  378.487587] CPU: 1 PID: 2277 Comm: insmod Tainted: G           OEL      6.2.0-mvp10v1+8-generic #mvp10v1+tdx
[  378.487843] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown unknown
[  378.488048] RIP: 0010:smp_call_function_many_cond+0x11e/0x3b0
[  378.488182] Code: 76 35 4c 63 e8 4c 8b 23 49 81 fd ff 1f 00 00 0f 87 4e 02 00 00 4a 8b 34 ed 80 7a e7 8b 4c 01 e6 8b 56 08 83 e2 01 74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb b1 48 83 c4 40 5b 41 5c 41 5d
[  378.488488] RSP: 0018:ff61389300cb7ae0 EFLAGS: 00000202
[  378.488601] RAX: 0000000000000006 RBX: ff4d95ab7fa72780 RCX: 0000000000000001
[  378.488800] RDX: 0000000000000001 RSI: ff4d95ab7fbb88a0 RDI: ff4d959db9466a88
[  378.488926] RBP: ff61389300cb7b48 R08: 0000000000000006 R09: 0000000000000000
[  378.489131] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000388a0
[  378.489313] R13: 0000000000000006 R14: 0000000000000001 R15: ff4d95ab7fa72780
[  378.489447] FS:  00007f8fea453000(0000) GS:ff4d95ab7fa40000(0000) knlGS:0000000000000000
[  378.489598] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  378.489782] CR2: 0000564dea5f05f0 CR3: 000000070c4aa006 CR4: 0000000000771ee0
[  378.489923] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  378.490119] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  378.490275] PKRU: 55555554
[  378.490325] Call Trace:
[  378.490402]  <TASK>
[  378.490455]  ? __pfx_do_flush_tlb_all+0x10/0x10
[  378.490551]  on_each_cpu_cond_mask+0x24/0x40
[  378.490660]  flush_tlb_kernel_range+0x41/0xc0
[  378.490738]  __purge_vmap_area_lazy+0xcf/0x780
[  378.490870]  ? purge_fragmented_blocks_allcpus+0x51/0x210
[  378.490984]  _vm_unmap_aliases+0x129/0x180
[  378.491119]  __vunmap+0x1a0/0x320
[  378.491215]  __vfree+0x22/0x70
[  378.491336]  vfree+0x2c/0x50
[  378.491450]  module_memfree+0x1c/0x30
[  378.491521]  load_module+0x763/0xc90
[  378.491601]  ? security_kernel_post_read_file+0x5c/0x70
[  378.491715]  ? kernel_read_file+0x24e/0x2a0
[  378.491806]  __do_sys_finit_module+0xc8/0x140
[  378.491910]  ? __do_sys_finit_module+0xc8/0x140
[  378.492011]  __x64_sys_finit_module+0x18/0x20
[  378.492092]  do_syscall_64+0x59/0x90
[  378.492263]  ? exit_to_user_mode_prepare+0x37/0xb0
[  378.492385]  ? syscall_exit_to_user_mode+0x26/0x50
[  378.492493]  ? __x64_sys_mmap+0x33/0x50
[  378.492606]  ? do_syscall_64+0x69/0x90
[  378.492702]  ? exit_to_user_mode_prepare+0x37/0xb0
[  378.492803]  ? irqentry_exit_to_user_mode+0x9/0x20
[  378.492902]  ? irqentry_exit+0x3b/0x50
[  378.492994]  ? exc_page_fault+0x87/0x180
[  378.493087]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  378.493191] RIP: 0033:0x7f8fe9d1e88d
[  378.493273] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 73 b5 0f 00 f7 d8 64 89 01 48
[  378.493648] RSP: 002b:00007fff79091e48 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[  378.493804] RAX: ffffffffffffffda RBX: 0000564deb44f490 RCX: 00007f8fe9d1e88d
[  378.493948] RDX: 0000000000000000 RSI: 0000564dea5f8cd2 RDI: 0000000000000003
[  378.494108] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[  378.494280] R10: 0000000000000003 R11: 0000000000000246 R12: 0000564dea5f8cd2
[  378.494446] R13: 0000564deb44e3d0 R14: 0000564dea5f7888 R15: 0000564deb44f5a0
[  378.494660]  </TASK>
[  388.254651] rcu: rcu_preempt kthread starved for 2475 jiffies! g7201 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=10
[  388.254910] rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[  388.255129] rcu: RCU grace-period kthread stack dump:
[  388.255247] task:rcu_preempt     state:I stack:0     pid:15    ppid:2      flags:0x00004000
[  388.255398] Call Trace:
[  388.255464]  <TASK>
[  388.255530]  __schedule+0x28d/0x5d0
[  388.255617]  ? __mod_timer+0x292/0x430
[  388.255681]  ? __pfx_rcu_gp_kthread+0x10/0x10
[  388.255798]  schedule+0x68/0x100
[  388.255882]  schedule_timeout+0x87/0x160
[  388.255975]  ? __pfx_process_timeout+0x10/0x10
[  388.256125]  rcu_gp_fqs_loop+0x115/0x450
[  388.256228]  rcu_gp_kthread+0xd0/0x160
[  388.256317]  kthread+0xeb/0x120
[  388.256386]  ? __pfx_kthread+0x10/0x10
[  388.256453]  ret_from_fork+0x29/0x50
[  388.256522]  </TASK>
[  388.256587] rcu: Stack dump where RCU GP kthread last ran:
[  388.256687] Sending NMI from CPU 12 to CPUs 10:
[  388.256812] NMI backtrace for cpu 10
[  388.256815] CPU: 10 PID: 0 Comm: swapper/10 Tainted: G           OEL      6.2.0-mvp10v1+8-generic #mvp10v1+tdx
[  388.256817] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown unknown
[  388.256818] RIP: 0010:__trace_tdx_hypercall+0x65/0x1d0
[  388.256823] Code: e8 e0 1b 00 00 48 8b 53 38 4c 8b 4b 30 49 89 c4 4c 8b 43 28 4c 8b 7b 20 4c 8b 73 18 66 90 48 8d 65 d8 4c 89 e0 5b 41 5c 41 5d <41> 5e 41 5f 5d c3 cc cc cc cc 65 8b 05 f6 ce a2 75 89 c0 48 0f a3
[  388.256824] RSP: 0018:ff613893000efdf0 EFLAGS: 00000246
[  388.256826] RAX: 0000000000000000 RBX: ff4d959db9b92000 RCX: 000000000000ffcc
[  388.256827] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
[  388.256828] RBP: ff613893000efe00 R08: 0000000000000000 R09: 0000000000000000
[  388.256828] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  388.256829] R13: 0000000000000000 R14: 000000000000000c R15: 0000000000000000
[  388.256830] FS:  0000000000000000(0000) GS:ff4d95ab7fc80000(0000) knlGS:0000000000000000
[  388.256831] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  388.256832] CR2: 00007f8a74000020 CR3: 0000001238410006 CR4: 0000000000771ee0
[  388.256833] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  388.256833] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  388.256834] PKRU: 55555554
[  388.256835] Call Trace:
[  388.256835]  <TASK>
[  388.256850]  tdx_safe_halt+0x3f/0xa0
[  388.256854]  arch_cpu_idle+0x12/0x20
[  388.256859]  default_idle_call+0x3d/0x100
[  388.256860]  cpuidle_idle_call+0x177/0x1d0
[  388.256865]  do_idle+0x82/0xf0
[  388.256867]  cpu_startup_entry+0x20/0x30
[  388.256869]  start_secondary+0x122/0x160
[  388.256872]  secondary_startup_64_no_verify+0xe5/0xeb
[  388.256876]  </TASK>
[  406.068777] NMI: PCI system error (SERR) for reason ff on CPU 6.
[  406.068780] Dazed and confused, but trying to continue
[  406.101940] ACPI Warning: \_SB.PCI0.S28.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20221020/nsarguments-61)
[  409.509042] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[  409.509238] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:1967
[  410.026820] NVRM: nvCheckOkFailedNoLog: Check failed: Call not supported [NV_ERR_NOT_SUPPORTED] (0x00000056) returned from pEntry->pCallback(pGpu, pEntry->pCallbackParam) @ kernel_fifo.c:3021

but boot continues..

Shortly after that i did the GPU Attestation

Number of GPUs available : 1
-----------------------------------
Fetching GPU 0 information from GPU driver.
Using the Nonce specified by user
VERIFYING GPU : 0
        Driver version fetched : 550.54.15
        VBIOS version fetched : 96.00.74.00.1c
        Validating GPU certificate chains.
                The firmware ID in the device certificate chain is matching with the one in the attestation report.
                GPU attestation report certificate chain validation successful.
                        The certificate chain revocation status verification successful.
        Authenticating attestation report
                The nonce in the SPDM GET MEASUREMENT request message is matching with the generated nonce.
                Driver version fetched from the attestation report : 550.54.15
                VBIOS version fetched from the attestation report : 96.00.74.00.1c
                Attestation report signature verification successful.
                Attestation report verification successful.
        Authenticating the RIMs.
                Authenticating Driver RIM
                        Fetching the driver RIM from the RIM service.
                        RIM Schema validation passed.
                        driver RIM certificate chain verification successful.
                        The certificate chain revocation status verification successful.
                        driver RIM signature verification successful.
                        Driver RIM verification successful
                Authenticating VBIOS RIM.
                        Fetching the VBIOS RIM from the RIM service.
                        RIM Schema validation passed.
                        vbios RIM certificate chain verification successful.
nvidia-persistenced failed to initialize. Check syslog for more details.
                        The certificate chain revocation status verification successful.
                        vbios RIM signature verification successful.
                        VBIOS RIM verification successful
        Comparing measurements (runtime vs golden)
                        The runtime measurements are matching with the golden measurements.
                GPU is in expected state.
        Setting the GPU Ready State to READY
        GPU 0 verified successfully.
        GPU Attested Successfully

NOTE the nvidia-persistenced failed to initialize. Check syslog for more details. which is part of the console and NOT the output of the GPU attestation

and finally

[  423.499159] NVRM: kgspCheckGspRmCcCleanup_GH100: CC secret cleanup successful!
[  425.483959] NVRM: gpumgrCheckRmFirmwarePolicy: Disabling GSP offload -- GPU not supported
[  425.484210] NVRM: nvAssertFailedNoLog: Assertion failed: GPU_GET_VGPU(pGpu) != NULL @ gpu.c:5282
[  425.484435] NVRM: nvAssertFailedNoLog: Assertion failed: pRpc != NULL @ objvgpu.c:150
[  425.484624] BUG: unable to handle page fault for address: 00000000000152ba
[  425.484776] #PF: supervisor read access in kernel mode
[  425.484900] #PF: error_code(0x0000) - not-present page
[  425.485026] PGD 5f2afc067 P4D 610805067 PUD 5f2af0067 PMD 0
[  425.485183] Oops: 0000 [#1] PREEMPT SMP NOPTI
[  425.485308] CPU: 6 PID: 2628 Comm: nvidia-persiste Tainted: G           OEL      6.2.0-mvp10v1+8-generic #mvp10v1+tdx
[  425.485557] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown unknown
[  425.485744] RIP: 0010:freeRpcInfrastructure_VGPU+0x25/0xe0 [nvidia]
[  425.486170] Code: ff eb 85 66 90 f3 0f 1e fa 55 48 89 e5 41 55 41 bd 40 00 00 00 41 54 53 48 83 ec 08 8b 87 e4 04 00 00 48 8b 1c c5 80 b4 d3 c0 <80> bb ba 52 01 00 00 74 5f 45 31 ed 80 bf 83 02 00 00 00 49 89 fc
[  425.486607] RSP: 0018:ff61389300a677c0 EFLAGS: 00010286
[  425.486732] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000020
[  425.486920] RDX: 0000000000000001 RSI: ff61389300a677b4 RDI: ff4d9593f2bac020
[  425.487113] RBP: ff61389300a677e0 R08: ffffffffc077fb38 R09: ff61389300a67608
[  425.487302] R10: 0000000000000000 R11: 0000001fd1a02000 R12: 0000000000000000
[  425.487488] R13: 0000000000000040 R14: 0000000000000000 R15: 00000000180000a1
[  425.487676] FS:  00007fc7cb4dcb80(0000) GS:ff4d95ab7fb80000(0000) knlGS:0000000000000000
[  425.487863] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  425.488019] CR2: 00000000000152ba CR3: 000000070c4aa001 CR4: 0000000000771ee0
[  425.488207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  425.488395] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  425.488584] PKRU: 55555554
[  425.488647] Call Trace:
[  425.488710]  <TASK>
[  425.488774]  vgpuDestructObject+0x5b/0x110 [nvidia]
[  425.489086]  gpuDestruct_IMPL+0x377/0x3e0 [nvidia]
[  425.489419]  __nvoc_dtor_OBJGPU+0x15/0x40 [nvidia]
[  425.489685]  __nvoc_objDelete+0x2c/0xf0 [nvidia]
[  425.489932]  gpumgrAttachGpu+0x912/0xea0 [nvidia]
[  425.490210]  RmInitAdapter+0x5ad/0x19f0 [nvidia]
[  425.490514]  ? _raw_spin_lock_irqsave+0xe/0x20
[  425.490643]  rm_init_adapter+0xad/0xc0 [nvidia]
[  425.490930]  nv_open_device+0x426/0xa30 [nvidia]
[  425.491158]  nvidia_open+0x224/0x530 [nvidia]
[  425.491386]  chrdev_open+0xc4/0x240
[  425.491481]  ? __pfx_chrdev_open+0x10/0x10
[  425.491575]  do_dentry_open+0x167/0x440
[  425.491670]  vfs_open+0x2d/0x40
[  425.491765]  do_open+0x219/0x4c0
[  425.491860]  ? open_last_lookups+0x97/0x3b0
[  425.491956]  path_openat+0x112/0x290
[  425.492050]  do_filp_open+0xb2/0x160
[  425.492145]  ? __check_object_size+0x1d/0x30
[  425.492270]  ? alloc_fd+0xa9/0x170
[  425.492365]  do_sys_openat2+0x9f/0x160
[  425.492460]  __x64_sys_openat+0x55/0x90
[  425.492554]  do_syscall_64+0x59/0x90
[  425.492649]  ? do_syscall_64+0x69/0x90
[  425.492742]  ? do_syscall_64+0x69/0x90
[  425.492836]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  425.492963] RIP: 0033:0x7fc7cb31453b
[  425.493058] Code: 25 00 00 41 00 3d 00 00 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 2b 14 25
[  425.493495] RSP: 002b:00007ffdd5ee2c90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[  425.493683] RAX: ffffffffffffffda RBX: 00007ffdd5ee2d30 RCX: 00007fc7cb31453b
[  425.493870] RDX: 0000000000080002 RSI: 00007ffdd5ee2d30 RDI: 00000000ffffff9c
[  425.494059] RBP: 00007ffdd5ee2d30 R08: 0000000000000000 R09: 00007ffdd5ee2b90
[  425.494248] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000080002
[  425.494435] R13: 00007ffdd5ee2e7c R14: 0000000000000002 R15: 00007fc7cb0611a0
[  425.494634]  </TASK>
[  425.494682] Modules linked in: veth(E) xt_tcpudp(E) xt_nat(E) xt_conntrack(E) xt_addrtype(E) nft_compat(E) nft_chain_nat(E) nft_masq(E) xt_MASQUERADE(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) br_netfilter(E) bridge(E) stp(E) llc(E) nf_tables(E) libcrc32c(E) nfnetlink(E) ip_tables(E) x_tables(E) nvidia_uvm(OE) nvidia(OE) aesni_intel(E) crypto_simd(E) cryptd(E) drm(E) ecdsa_generic(E) ecdh_generic(E) ecc(E) e1000(E) virtio_scsi(E) vmw_vsock_virtio_transport(E) vhost_vsock(E) vmw_vsock_virtio_transport_common(E) vsock(E) vhost(E) vhost_iotlb(E) dm_integrity(E) dm_bufio(E) async_xor(E) async_tx(E) xor(E) essiv(E) authenc(E) dm_crypt(E)
[  425.496025] CR2: 00000000000152ba
[  425.496119] ---[ end trace 0000000000000000 ]---
[  425.496242] RIP: 0010:freeRpcInfrastructure_VGPU+0x25/0xe0 [nvidia]
[  425.496579] Code: ff eb 85 66 90 f3 0f 1e fa 55 48 89 e5 41 55 41 bd 40 00 00 00 41 54 53 48 83 ec 08 8b 87 e4 04 00 00 48 8b 1c c5 80 b4 d3 c0 <80> bb ba 52 01 00 00 74 5f 45 31 ed 80 bf 83 02 00 00 00 49 89 fc
[  425.497016] RSP: 0018:ff61389300a677c0 EFLAGS: 00010286
[  425.497141] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000020
[  425.497328] RDX: 0000000000000001 RSI: ff61389300a677b4 RDI: ff4d9593f2bac020
[  425.497516] RBP: ff61389300a677e0 R08: ffffffffc077fb38 R09: ff61389300a67608
[  425.497702] R10: 0000000000000000 R11: 0000001fd1a02000 R12: 0000000000000000
[  425.497890] R13: 0000000000000040 R14: 0000000000000000 R15: 00000000180000a1
[  425.498077] FS:  00007fc7cb4dcb80(0000) GS:ff4d95ab7fb80000(0000) knlGS:0000000000000000
[  425.498262] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  425.498420] CR2: 00000000000152ba CR3: 000000070c4aa001 CR4: 0000000000771ee0
[  425.498607] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  425.498793] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  425.498981] PKRU: 55555554
[  425.499052] Kernel panic - not syncing: Fatal exception
[  425.499498] Kernel Offset: 0x9600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  425.499742] ---[ end Kernel panic - not syncing: Fatal exception ]---
Tan-YiFan commented 1 month ago

@YurkoWasHere Could you attach the kernel config file of the minimalist OS (seemingly 6.2.0-mvp10v1+8-generic #mvp10v1+tdx)?

YurkoWasHere commented 3 weeks ago

Its a straight copy from the ubuntu build from the docs.

Tan-YiFan commented 3 weeks ago

In ubuntu config, some crypto-related features are marked as "m", such as CONFIG_CRYPTO_AES_NI_INTEL=m. Please check that those modules are properly installed in the guest VM.

rnertney commented 2 weeks ago

I think As CC is wiped the H100 becomes uninitialized in the CC environment.?

Yes, that is correct. We don't wipe until the FLR comes, but if the driver is unloaded, the shared secrets are wiped by design.

Its a straight copy from the ubuntu build from the docs.

You are using this doc?. This build should automatically handle all kconfigs

Pay attention to the nvidia-persistenced --uvm-persistence-mode before running any commands.

You can also set the GPU into cc-off mode from the host with nvidia_gpu_tools.py , and boot the TDX VM into non-CC mode via sudo ./start-qemu.sh -i build/ubuntu-22.04/guest-image/td-guest-ubuntu-22.04.qcow2 -b grub -t efi

Once you boot a totally traditional VM, run nvidia-smi -q | grep VBIOS and let us know the VBIOS version.