How to run programs on multiple machines, such as 4 machines with 8 gpus(4*8=32 in total)?

dandelin / ViLT

Code for the ICML 2021 (long talk) paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

Apache License 2.0

1.36k stars 209 forks source link

How to run programs on multiple machines, such as 4 machines with 8 gpus(4*8=32 in total)? #46

Closed raojay7 closed 2 years ago

raojay7 commented 2 years ago

export MASTER_ADDR=$DIST_0_IP export MASTER_PORT=$DIST_0_PORT export NODE_RANK=$DIST_RANK python run.py with data_root= num_gpus= num_nodes= task_mlm_itm whole_word_masking=True step200k per_gpu_batchsize=

ex) python run.py with data_root=/data2/dsets/dataset num_gpus=8 num_nodes=1 task_mlm_itm whole_word_masking=True step200k per_gpu_batchsize=64

How do I set up these codes and what other operations need to be done？