The validation set potentially can be much larger than a minibatch. Currently we use the same method for calculating loss/acc for both minibatch training set and validation set.
This can cause severe memory problems since with a large LSTM and a large validation set, the tensors do not fit into memory any more. The tensors also would not fit into the GPU any more.
Instead, make sure that the validationset is evaluated using the same mini batch size as for training.
Also, be clever about whether or not to load and hold the whole validation set into memory or re-read for each validation.
The validation set potentially can be much larger than a minibatch. Currently we use the same method for calculating loss/acc for both minibatch training set and validation set.
This can cause severe memory problems since with a large LSTM and a large validation set, the tensors do not fit into memory any more. The tensors also would not fit into the GPU any more.
Instead, make sure that the validationset is evaluated using the same mini batch size as for training. Also, be clever about whether or not to load and hold the whole validation set into memory or re-read for each validation.