Optimizer drops out [here, worker 9 out of 12 optimizers] with below error. RMQ is fine.
The other optimizers then drop out:
Traceback (most recent call last):
File "optimizer.py", line 353, in <module>
pretrained_model=args.pretrained_model,
File "optimizer.py", line 335, in main
dota_optimizer.run()
File "optimizer.py", line 171, in run
self.step(experiences=experiences)
File "optimizer.py", line 205, in step
loss = self.finish_episode(rewards=all_discounted_rewards, log_probs=all_logprobs)
File "optimizer.py", line 118, in finish_episode
loss.backward()
File "/root/.local/lib/python3.7/site-packages/torch/tensor.py", line 102, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/root/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 90, in backward
allow_unreachable=True) # allow_unreachable flag
File "/root/dotaclient/distributed.py", line 37, in allreduce_params
dist.all_reduce(has_grad_count, op=dist.ReduceOp.SUM) # [0. to world_size]
File "/root/.local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 838, in all_reduce
work.wait()
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:543] Connection closed by peer [10.20.94.2]:15019
Optimizer drops out [here, worker 9 out of 12 optimizers] with below error. RMQ is fine.![image](https://user-images.githubusercontent.com/7721540/50739880-e8ce9800-119a-11e9-9651-918f01008dd0.png)
The other optimizers then drop out: