Closed rondogency closed 3 years ago
This issue only occurs when mixing the conda-toolchain python interpreter with system toolchain mpi. As a workaround, when using conda make sure to either build horovod mpi with the conda mpi or stop using conda. Ie ensure that the toolchain matches.
Description
BERT Pretraining with Horovod hangs intermittently in the middle of the training, this happens around 1 out of 4 runs and hangs at mostly around 4400 steps
Currently I can only reproduce it with Horovod CPU mode built with HOROVOD_GPU_OPERATION=MPI, but cannot reproduce with HOROVOD_GPU_OPERATION=NCCL
More inspection shows that the hangs happens in one of the rank process the datasetloader creating a lot of threads, causing that process to hang, and other process will wait for that process causing the whole training to hang
Error Message
From mpirun output error message regarding the hanging process
If putting error logs, at the time of hanging the batchify worker pool function won't be run https://github.com/dmlc/gluon-nlp/blob/v0.10.x/src/gluonnlp/data/datasetloader.py#L55
To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
Steps to reproduce
(Paste the commands you ran that produced the error.)
What have you tried to solve it?
Environment
Horovod=0.20.0 with HOROVOD_GPU_OPERATION=MPI gluonnlp==0.10.0 mxnet_cu101==1.6.0