search

NVIDIA / nccl

Optimized primitives for collective multi-GPU communication
Other
3.24k stars 817 forks source link

Is point to point halo communication 'null peer' possible? #1416

Open huccpp opened 2 months ago

huccpp commented 2 months ago

Hello. In halo communication in MPI for topology non periodic, let's say 1D, the first and the last rank only communicate to one side of the neighbor. For the other side without neighbor, we can the put the rank source (recv from) and destination (send to) be MPI_PROC_NULL and the communication will not be initiated. As a result, for every rank we can simiply write MPI send and receive without special treatment (add a if statement) for the first and last rank in halo. Is there similar thing for nccl halo communication? Or we have to do a if statement? Thanks.

sjeaugey commented 2 months ago

There is no equivalent in NCCL; you need to add if statements indeed.

huccpp commented 2 months ago

Thanks for your reply sjeaugey.