Multi node training

[ADAM-CT]

I hope to add examples of multi node training

[ADAM-CT]

I wonder if there is something wrong with my method? Multi-machine pipedream can hardly be successfully trained, and it will report an error every time. As the first few questions I raised, can anyone help me

[deepakn94]

Not sure what your method is. We were able to successfully run multi-machine PipeDream experiments.

Some instructions are here: https://github.com/msr-fiddle/pipedream/blob/master/EXPERIMENTS.md. In particular, look for the *_16gpu.yml files, which are all run using multiple servers. Alternatively, please send me the commands you're trying to run, and I can try helping you out.

[ADAM-CT]

thank you for your reply! server1: python -m launch --nnodes 2 --node_rank 0 --nproc_per_node 8 main_with_runtime.py --data_dir ../../data --master_addr 172.17.2.1 --module models.vgg16.gpus=16 --distributed_backend gloo -b 64 --lr 0.040000 --lr_policy polynomial --weight-decay 0.000500 --epochs 20 --print-freq 100 --verbose 100 --num_ranks_in_server 8 --config_path models/vgg16/gpus=16/hybrid_conf.json

server2: python -m launch --nnodes 2 --node_rank 1 --nproc_per_node 8 main_with_runtime.py --data_dir ../../data --master_addr 172.17.2.1 --module models.vgg16.gpus=16 --distributed_backend gloo -b 64 --lr 0.040000 --lr_policy polynomial --weight-decay 0.000500 --epochs 20 --print-freq 100 --verbose 100 --num_ranks_in_server 8 --config_path models/vgg16/gpus=16/hybrid_conf.json

[ADAM-CT]

>Note: The models used next are from your generated models, I just use them to train

when i use --config_path models/vgg16/gpus=16_straight/hybrid_conf.json

it will throw the following error: ValueError: optimizer got an empty parameter list Traceback (most recent call last): File "main_with_runtime.py", line 578, in main() File "main_with_runtime.py", line 229, in main macrobatch=args.macrobatch) File "../sgd.py", line 23, in init macrobatch=macrobatch, File "../optimizer.py", line 41, in init master_parameters, **optimizer_args) File "/opt/conda/lib/python3.6/site-packages/torch/optim/sgd.py", line 64, in init super(SGD, self).init(params, defaults) File "/opt/conda/lib/python3.6/site-packages/torch/optim/optimizer.py", line 45, in init raise ValueError("optimizer got an empty parameter list") ValueError: optimizer got an empty parameter list

when i use --config_path models/vgg16/gpus=16/mp_conf.json

it will throw the following error:

rank_to_stage_map {0: 0, 1: 1} Traceback (most recent call last): File "main_with_runtime.py", line 578, in main() File "main_with_runtime.py", line 191, in main enable_recompute=args.recompute) File "../runtime.py", line 64, in init master_addr, rank, local_rank, num_ranks_in_server) File "../runtime.py", line 150, in initialize assert 0 <= self.rank < len(rank_to_stage_map) AssertionError

but when i use --config_path models/vgg16/gpus=16/hybrid_conf.json

it will successful!!!

So I am very confused, why is there such a result, my environment has not changed in any way, just changed the configuration file

my enviroment: server 1: 8 V100
server2: 8 V100

[ADAM-CT]

I think in theory, taking your model directly can at least train successfully, but something went wrong

[ghost]

If I understand correctly, there are some problems in models/vgg16/gpus=16/mp_conf.json. This configuration file only contains rank 0 and rank 1, which conflicts with the setting of gpu=16 (should contain rank 0 to rank 15).

When I use models/vgg16/gpus=16_straight/hybrid_conf.json, I encountered the same problem with @ADAM-CT . After observing the error message, I found that only certain processes had problems (process 0, 2, 4, 6). So I check the corresponding stage*.py of these processes in models/vgg16/gpus=16_straight/ and I found that the commonality of these stages is that none of them contain the layers with trainable parameters.

It works well after I modify the partition and let each stage have at least one layer with trainable parameters.

I think the reason for this problem should be that the current program cannot handle certain stages that do not contain trainable parameters.