Closed asdfry closed 1 month ago
Could you try with a newer NCCL version (preferably the latest 2.22.3)? There was a packet reordering issue during bootstrap that we've fixed in 2.21, which would result in Message truncated
errors like what you observe on Servers 2 and 3.
If that doesn't help, then we'll need to see the NCCL output from all three servers obtained when running NCCL with NCCL_DEBUG=INFO
.
I solved it with the command below. Thank you for your response.
Server 1: NCCL_NVLS_ENABLE=0 torchrun --nnodes=3 --nproc_per_node=8 --node_rank=0 --master_addr=10.244.2.207 --master_port=1040 train.py -b 1 -dn tldr_news -m bloom-560m
Description
Hello, I am testing multi-node training with three servers, each equipped with different GPUs (H1008, A404, L40S*4). During the process, I encountered an NCCL error and seek assistance to resolve this issue.
Environment
NCCL version: 2.20.5 PyTorch version: 2.3.1 CUDA version: 12.3 Operating System: Ubuntu 22.04.3 LTS Driver version: 535.161.08 Network Configuration: Kubernetes CNI
Commands Executed
Server 1:
torchrun --nnodes=3 --nproc_per_node=8 --node_rank=0 --master_addr=10.244.2.207 --master_port=1040 train.py -b 1 -dn tldr_news -m bloom-560m
Server 2:torchrun --nnodes=3 --nproc_per_node=4 --node_rank=1 --master_addr=10.244.2.207 --master_port=1040 train.py -b 1 -dn tldr_news -m bloom-560m
Server 3:torchrun --nnodes=3 --nproc_per_node=4 --node_rank=2 --master_addr=10.244.2.207 --master_port=1040 train.py -b 1 -dn tldr_news -m bloom-560m
Error Logs
Server 1:
Server 2, 3:
Additional Context
Even after changing the NCCL version and the PyTorch version, the same error occurred. I would greatly appreciate any guidance or suggestions to resolve this issue. Thank you.