Closed liaoweiguo closed 7 years ago
Trying reducing your batch size from 16 to 8 (or even lower) or reducing your max input duration (or a combination of these things).
The reason the speech recognition model might have OOM CUDA errors come up sporadically is that the memory taken up by the model depends on the number of input audio files and their lengths for each batch. If on the first epoch still, then this will tend to happen near the end of the epoch where the longer files are.
Thanks for quick response. I deploy on another machine and it works, I will test different para later. How many epochs will provide acceptabe models, 500, 5000?
and will you privide deepspeech2 model?
Acceptable results can be obtained after around 5-50 epochs, depending on your dataset size. We'll stay quiet about Deepspeech 2 for now :)
DeepSpeech model I use a Quadro M2000 4G GPU, tf failed the trainning for out of GPU memory thanks a lot