search

namisan / mt-dnn

Multi-Task Deep Neural Networks for Natural Language Understanding
MIT License
2.24k stars 412 forks source link

I want to know what is the use of argument “local_rank” in training #208

Closed XuLongjia closed 2 years ago

XuLongjia commented 3 years ago

parser.add_argument("--local_rank", type=int, default=-1, help="For distributed training: local_rank") I don‘t know it’s meaning and what role playing in model training， Looking forward to your answer！ Thanks！

namisan commented 3 years ago

It is used for DDP distributed training which gives a great speed up. I'll update the read me.

namisan commented 2 years ago

Please refer the script. Hope it helps. https://github.com/namisan/mt-dnn/blob/master/experiments/glue/run_glue_finetuning.sh#L112