search

NVIDIA / gdrcopy

A fast GPU memory copy library based on NVIDIA GPUDirect RDMA technology
MIT License
898 stars 144 forks source link

Add a new API to get DMA map of memory #304

Closed ajitkhaparde closed 2 months ago

ajitkhaparde commented 2 months ago

Add a new API to get the DMA memory or the Physical Address of the peer mapped memory. This will allow NIC devices to use the physical addresses directly instead of using the CUDA provided Unified Memory Address.

pakmarkthub commented 2 months ago

Hi @ajitkhaparde,

If my understanding is correct, gdr_p2p_dma_map_buffer is the same as gdr_pin_buffer. However, it also returns paddr of the cudaMalloc'd VA. Why do you need this information? Will you pass this paddr to your NIC stack?

At least for NVIDIA NICs, nvidia-peermem driver is used to provide this functionality. That PA and the corresponding DMA addresses are exchanged inside the kernel space. You can also use DMABUF instead of nvidia-peermem driver.

ajitkhaparde commented 2 months ago

Yes. That is correct. I am using that in the DPDK gpudev library. At the end of it all, the PA is passed to the NIC driver for direct DMA to the GPU memory. Here is the patchwork link. https://patches.dpdk.org/project/dpdk/list/?series=32920

Thanks Ajit

On Thu, Sep 5, 2024 at 9:24 PM Pak Markthub @.***> wrote:

Hi @ajitkhaparde https://github.com/ajitkhaparde,

If my understanding is correct, gdr_p2p_dma_map_buffer is the same as gdr_pin_buffer. However, it also returns paddr of the cudaMalloc'd VA. Why do you need this information? Will you pass this paddr to your NIC stack?

At least for NVIDIA NICs, nvidia-peermem driver is used to provide this functionality. That PA and the corresponding DMA addresses are exchanged inside the kernel space. You can also use DMABUF instead of nvidia-peermem driver.

— Reply to this email directly, view it on GitHub https://github.com/NVIDIA/gdrcopy/pull/304#issuecomment-2333208109, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF5LBLEMC7OGIEWGVMIBQMLZVEVBRAVCNFSM6AAAAABNXYQGM2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZTGIYDQMJQHE . You are receiving this because you were mentioned.Message ID: @.***>

drossetti commented 2 months ago

FYI exposing phys addresses to user space is considered a security risk, hence this MR would not pass a security review. Depending on your use case, it might still be usable for you.

pakmarkthub commented 2 months ago

Exposing PA to user space is considered a security risk. We cannot accept this change. Closing it.