Memory allocation error on GPU

[ymcui]

Hi,

Recently, I used Keras to do some language modeling with TESLA K40M GPU. I set up an LSTM-LM, and it works well for a small data set (~1M in txt format), and GPU memory is only consumed for approximately ~300M. But when i tried to train on a larger data set (~150M in txt format), the program can not be executed, and report "Memory Allocation Error". it indicated that the program wants to allocate >30GB GPU memory space, which is impossible for any regular GPUs. The hyper-parameters for small and large datasets are identical(hiddenSize=100,vocabulary=30k). I don't know whether the memory allocation is also related to the input data size or not.

Thanks in advance.

[jocicmarko]

You could try reducing batch_size in training and predicting, at least that helped me with big models and big datasets.

[ymcui]

@jocicmarko thank you, i will try that.