search

linux-rdma / perftest

Infiniband Verbs Performance Tests
Other
615 stars 290 forks source link

Comparison of GPUDirect ( GPU Memory)- NIC access vs Hop Memory GPU-> Host Memory -> NIC #297

Open alokprasad opened 4 days ago

alokprasad commented 4 days ago

Currently perftest supports GPU Direct support where NIC can directly access GPU memory , but it would be good to have comparison it without GPU Direct i.e. GPU Memory -> Copied to Host Memory -> NIC . Can someone give pointer how to make this change. what i think we need to allocate host memory and copy gpu memory using cuMemcpyDtoH , then this host memory need to be used for MR?

sshaulnv commented 3 days ago

Hi @alokprasad, do you mean to perform the copies in the datapath?

alokprasad commented 3 days ago

@sshaulnv yes..that would give insight on the improvement achieved by gpudurect

sshaulnv commented 3 days ago

To ensure optimal bandwidth, we generally avoid performing intensive operations within the datapath. Assuming GPUDirect is unavailable and we need to send a message from GPU memory, we would probably first copy the buffer to host memory before entering the datapath.

alokprasad commented 2 days ago

@sshaulnv i agree thats good solution if we have constant data..but consider a scenario that Host 1 sends GPU data to Host 2 and it recives back Host2 does some processing , we need to do constant copying host mem-gpu mem in data path.