Using distributed or parallel set-up in script?: ddp
Who can help?
@muellerzr and @pacman100
Information
[ ] The official example scripts
[X] My own modified scripts
Tasks
[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)
Reproduction
Use DDP to trigger the training script
torchrun --standalone --nnodes=1 --nproc-per-node=$NUM_GPUS train.py --config /opt/ml/input/config/hyperparameters.json
In trainer argument set load_best_model_at_end to true
At the end of the script, all GPUs except the rank 0 emmit the following error
RuntimeError: DDP expects same model across all ranks, but Rank 5 has 128 params, while rank 0 has inconsistent 1506656875 params.
return dist._verify_params_across_processes(process_group, tensors, logger)
return dist._verify_params_across_processes(process_group, tensors, logger)RuntimeError
System Info
transformers
version: 4.40.1Who can help?
@muellerzr and @pacman100
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
torchrun --standalone --nnodes=1 --nproc-per-node=$NUM_GPUS train.py --config /opt/ml/input/config/hyperparameters.json
Expected behavior
No Error occur.