search

AMDESE / AMDSEV

AMD Secure Encrypted Virtualization
304 stars 87 forks source link

Using RDMA with AMD SEV/SME enabled #51

Open aakahlow opened 4 years ago

aakahlow commented 4 years ago

I wonder if it is possible to use RDMA natively or inside VMs with AMD SEV/SME ? And if it will require any specific system configuration ?

Thanks

tlendacky commented 4 years ago

Let me start by saying that I don't have a lot of experience with RDMA and so I'm not sure if it uses the Linux kernel DMA APIs.

Since RDMA transfers directly to userspace buffers, which will be mapped encrypted, then:

For SME (without IOMMU enabled): The device must be programmed with DMA addresses that have the encryption bit set. If not, then the DMA will be performed un-encrypted, but accessed by the application as encrypted. So the application will see ciphertext.

For SME (with IOMMU enabled): The IOMMU page tables must set the encryption bit in the page table entry for the IOVA using for the DMA address. If not, the same thing mentioned above in the non-IOMMU case will happen.

For SEV: The device can only DMA to shared (unencrypted) pages, which then have to be copied to their final destination. Again, if the DMA is not to a shared page, then the data will look like ciphertext to the application.

In Linux, this is all handled under the covers by the DMA APIs. When the IOMMU is not enabled, the DMA APIs will automatically provide the DMA address with the encryption bit if the device and driver support DMA addresses where the encryption bit lives (currently bit 47). Otherwise the DMA subsystem will bounce the transfer using SWIOTLB. When the IOMMU is enabled, the IOMMU page tables are created with the encryption bit set in the page table entry.

There is no IOMMU in an SEV guest, so SWIOTLB is always used, except for dma_alloc_coherent(), which will allocate memory and then perform a set_memory_decrypted() to clear the encryption bit.

aakahlow commented 4 years ago

Hi Tom,

Sorry for late response here.

I have a related question. The rdma driver's code seem to be using the Linux Kernel's DMA APIs. I wonder, if the device driver needs to explicitly tell the DMA APIs to enable encryption bit?

Thanks for your help!

-Ayaz

tlendacky commented 4 years ago

Sorry, I just don't know enough about how RDMA functions relative to DMA and DMA buffers. I think you probably just need to try it. If it doesn't work, you'll need to instrument the code to see what is going on and why it is failing.