sjeaugey
commented
2 years ago Extracting the tree info:
como:60725:60748 [0] NCCL INFO Trees [0] 3/-1/-1->0->-1 [1] -1/-1/-1->0->3 [2] 3/-1/-1->0->-1 [3] -1/-1/-1->0->3 [4] 3/-1/-1->0->-1 [5] -1/-1/-1->0->3 [6] 3/-1/-1->0->-1 [7] -1/-1/-1->0->3
como:60725:60749 [1] NCCL INFO Trees [0] -1/-1/-1->1->2 [1] 2/-1/-1->1->-1 [2] -1/-1/-1->1->2 [3] 2/-1/-1->1->-1 [4] -1/-1/-1->1->2 [5] 2/-1/-1->1->-1 [6] -1/-1/-1->1->2 [7] 2/-1/-1->1->-1
como:60725:60750 [2] NCCL INFO Trees [0] 1/-1/-1->2->3 [1] 3/-1/-1->2->1 [2] 1/-1/-1->2->3 [3] 3/-1/-1->2->1 [4] 1/-1/-1->2->3 [5] 3/-1/-1->2->1 [6] 1/-1/-1->2->3 [7] 3/-1/-1->2->1
como:60725:60751 [3] NCCL INFO Trees [0] 2/-1/-1->3->0 [1] 0/-1/-1->3->2 [2] 2/-1/-1->3->0 [3] 0/-1/-1->3->2 [4] 2/-1/-1->3->0 [5] 0/-1/-1->3->2 [6] 2/-1/-1->3->0 [7] 0/-1/-1->3->2We have 8 channels. The first one is:
[0] 3/-1/-1->0->-1
[0] -1/-1/-1->1->2
[0] 1/-1/-1->2->3
[0] 2/-1/-1->3->0Given -1 means "no rank", it means the tree goes through the GPUs with this order: 1->2->3->0. You can do the same for the other channels, or you can also dump the graph XML and look at it; it should be consistent with that order.
The lines:
NCCL INFO Channel 01/08 : 0 3 2 1represent the rings, which are rotated to always start at the local rank. Since rank 0 prints the rings, they all start at 0, but those are rings, there is no start or end.
XiaoSong9905
RickyShi46
Dear NCCL Developer,
I'm confused about the Tree topology used in the 4 GPU DGX1-V100 (GPU 0,1,2,3) algorithm. My topology file looks like this
The NCCL INFO output when running allreduce & set NCCL_ALGO=Tree is
I don't quite understand what do
NCCL INFO Trees [0] ...part represent. How do channelNCCL INFO Channel 01/08 : 0 3 2 1's tree topology looks like? 0 -> 3, 3 -> 2, 3->1 ? So every tree start with root at 0?Thanks you so much for help.
Xiao