search

karpathy / char-rnn

Multi-layer Recurrent Neural Networks (LSTM, GRU, RNN) for character-level language models in Torch
11.63k stars 2.59k forks source link

Out of memory error while evaluating a split #96

Open anjishnu opened 9 years ago

anjishnu commented 9 years ago

I was trying to train a model on a fairly large corpus, 6GB for text (0.98% training, 0.02% validation)

Luckily this Pull Request:

Solved most of my data problems - the only issue arrived during the periodic validations that were running caused my training to error out.

I think the eval split function needs to be modified as well to support chunked evaluation. https://github.com/karpathy/char-rnn/blob/master/train.lua#L186

I'm new to Lua so I still haven't been able to figure out which part of the code is the memory bottleneck in this function.

karpathy commented 9 years ago

Hmmm, two things:

  1. that PR (#74) was merged to master a while ago, so it's odd that you had to merge it manually
  2. the eval() function does not actually do any reading from files, so it's odd that it would crash. Not sure what is going on here.