smatzek
commented
4 years ago The TensorFlow LMS version you are using (TFLMSv2 in WML CE 1.6.2) is a closed source implementation and its code is not in this repository. See here.
I would highly recommend updating to the new Large Model Support for TensorFlow 2 implementation which is open source in the master branch of this repository. This implementation is also available from the WML CE early access channel, and it is built into the TensorFlow 2.1.0 GPU package there. This link has more information about the WML CE early access channel. You should be able to install this version on Summit. The documentation on how to enable and use this version of LMS is in this github repository.
The issue you are hitting with TFLMSv2 is most likely an issue we recently found with TFLMSv2 and modern versions of Horovod. Earlier versions of Horovod from last year did not hit this. Horovod changed how it limits the GPU that each of the distributed processes sees. It now uses TensorFlow's GPU limiting functionality. Which part of the process touches the GPU first (LMS vs Horovod vs other TensorFlow code) now matters.
In your code you are using LMS auto-tuning which touches the GPU to find its memory capacity. To workaround this issue you should either specify values for the swapout_threshold, swapin_groupby, and swapin_ahead parameters, which avoids the auto-tuning, or you can explicitly set the memory auto tuning level using the autotune_gpu_mem LMS property.
So either:
lms_hook = LMS(swapout_threshold=1, swapin_groupby=0, swapin_ahead=1) # These are the max swapping, slowest data throughput parameters. Adding sync_mode=3 would also allow for higher amount of data.or
lms_hook = LMS()
lms.autotune_gpu_mem = 15 # For using 15 GB of GPU memory to leave a bit for base overhead usage.On the topic of BERT, @FarrandTom has trained BERT large on 16GB GPUs using the TFLMSv2 version. He wrote this post about it: https://medium.com/systems-ai/the-consequence-of-modern-nlp-approaches-647b2cabc5ec
I would suggest reading that article as he mentions which LMS tuning parameter values he used.
bethune-bryant
agemagician
Hello,
I am training bert model on SUMMIT using Horovod, DDL and ibm-wml-ce/1.6.2-2 , and it is working without an issue. However, when I tried to enable LMS it always crash as follows:
This is how I did modify the code:
Any idea how to solve this issue?