Closed slyviacassell closed 3 years ago
Yep, if you use multi-node training, the node_rank
is different for the specific node, and I illustrate to change this in the README.md
It needs you to modify manually because when using NCCL, you have to launch each script in each node manually in PyTorch, and one can never assign the node rank you specify in the cluster. In this way, it‘s better not to take this argument into the scripts. Or, you can give your advice, you're welcome.
Oww, my fault. I will close this issue. Thank you for your prompt reply!
Thank you for using my scripts~ Any questions are welcome~
Hi, I find that maybe your distributed bash scripts specify a wrong value for
node_rank
. This code may be wrong for launching the pytorch distributed training program. Thenode_rank
would be different for each node. But I find that your multinodes bash script does not change it.