search

NVIDIA / sentiment-discovery

Unsupervised Language Modeling at scale for robust sentiment classification
Other
1.06k stars 202 forks source link

Distributed training with Horovod? #58

Open Qiuzhuang opened 5 years ago

Qiuzhuang commented 5 years ago

Can we have the Distributed training with Horovod version -- we want to speed up LM training via a cluster of GPU machines.

raulpuric commented 5 years ago

We do not use horovod for pytorch distributed training. We use our own multinode launchers that adhere to pytorch's distributed training format for multinode training. Pytorch's multinode format is mpi-like and fairly similar to horovod so you may be able to use it, but I can't say for sure. Here's a more updated example of PyTorch distributed training for language modeling. You should be able to set similar environment variables to run multinode training.