Closed Feynman27 closed 7 years ago
This usually means that the GPU wasn't able to allocate enough memory to fit the mini-batch. It was fine for the first few utterances, because we sort the utterances by their duration in the first epoch (so the mini-batches were in general smaller in size). You can either try decreasing the size of the mini-batch or use a different GPU that has more memory (I'm not sure how the K520 manages its memory - it has two 4GB memory areas). If this doesn't help, also try reducing the number of layers or number of neurons in each layer. Most of the memory requirement is because we try to store activations through time.
Thanks, this makes sense. Reducing the batch size seems to be working.
This happened to me on 4GB and 8GB GPUs. Disabling sorting by duration solved the issue, otherwise memory will be exhausted in first epoch. I'm curious to know the reason. I tried on sorted dataset and when I disabled minibatch shuffling, It saturated memory again.
Hi,
I'm running this model using a Docker image on an Amazon EC2 instance with a GRID K520 gpu. The data is the "clean" data set from http://www.openslr.org/12/. Everything seems to work fine until iteration 60, after which I hit a MemoryError that I don't quite understand. The trace is provided below: