Open BlakeLazarine opened 2 years ago
Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue. Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly. If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.
Description
(A clear and concise description of what the bug is.) When running distributed training (multi-instance with each instance having a single GPU) with sparse gradients (produced by negative sampling), MXNet crashes. I have implementations that use either Synchronous or Asynchronous Parameter server implementations or use Horovod for distributed training. All of these implementations are able to train on datasets which do not use sparse gradients and the Horovod implementation successfully trains when there is only 1 instance.
The Async case seems to be the most descriptive of the issue, but I am still unable to make use of it.
Error Message
Error message from Async PS implementation
MXNetError: Check failed: static_cast<size_t>(pskv.size) == pskv_size (172770864 vs. 447000596) : The value size cannot be changed 447000596. Key is 3
Error message from Horovod Implementation
The Sync PS implementation does not crash, but has very high loss.
To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
PS Async
Horovod:
Only difference is in how the trainer is created
The stepping is done as:
The sparse nature of the data is passed through the nn.Embedding block and into the loss function.
I am withholding parts of the code for the purpose of data security
Steps to reproduce
(Paste the commands you ran that produced the error.)
Using AWS Sagemaker to run multi-instance training, launched with toolkit https://github.com/aws/sagemaker-mxnet-training-toolkit/tree/2f26babd9ba72f48d2336f7817d8255b6b2a2adc/src/sagemaker_mxnet_container
Use dataset with sparse gradients (very large vocabulary size). Note that training is fine on this dataset when only a single instance is used3. 3.
What have you tried to solve it?
Change MXNet versions (downgraded to 1.8)
Try 3 different approaches to distributed training
Use logging to identify breaking point.
Attempt to understand the back-end implementation of kvstore_dist.h, but was unable to make sense of the error message line.
Environment
MXNet 1.9.0 Horovod 0.19.0 OpenMPI 4.0.1 cuda 11.2.2 protobuf==3.20.1 \ h5py==2.10.0 \ onnx==1.8.1 \ "numpy<1.20" \ pandas==1.3.0 \ "Pillow>=9.0,<10.0" \ "requests<3" \ scikit-learn \ scipy==1.7.0 \ gluonnlp==0.10.0 \ gluoncv==0.8.0 \ "urllib3<2" \ python-dateutil==2.8.0 \ tqdm==4.39.0 \ "PyYAML>=5.4,<5.5" \ mpi4py==3.0.3 \ ${MX_URL} \ awscli \ s3fs==0.4.2 \ opencv-python Instance type: AWS ml.g4dn.xlarge