Open jxh314 opened 3 months ago
Yes, the data will be evenly distributed across various channels. The buffSize of each channel is controlled by an environment called "NCCL_BUFFSIZE" : https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html?highlight=nccl_buffsize
Thank you for your response. May I ask in which function or .cc file is the total transmission data evenly distributed among the channels implemented? Could you kindly point it out if possible? @visualxu
While fine-tuning the Opt-1.3b model using DeepSpeed across two nodes—each node having two RoCE network interfaces, namely mlx5_0 and mlx5_1, I've configured NCCL to utilize the ring algorithm and inspected RDMA packets via tcpdump on particular interfaces. My findings indicate that the amount of data transmitted on every channel seems to be approximately equal. Does this imply that data is uniformly distributed among distinct channels, or is this merely a coincidence?
Additionally, if feasible, could you point me to the source code segment responsible for data distribution across channels for my perusal?