increase in memory usage as training progresses

ottokart / punctuator2

A bidirectional recurrent neural network model with attention mechanism for restoring missing punctuation in unsegmented text

http://bark.phon.ioc.ee/punctuator

MIT License

659 stars 195 forks source link

increase in memory usage as training progresses #15

Closed Laqshay closed 6 years ago

Laqshay commented 7 years ago

I am training the model on an AWS Server (Ubuntu 16.04, NVIDIA Tesla K80). The memory (CPU RAM) is slowly used up as the training progresses, and I am not even able to complete 200 iterations before the process is killed due to the server running out of memory. The training runs fine on my laptop (CPU), but with a speed of ~150 samples per second.

I would like to run this on the GPU server. Could you please help me with what is causing this and how to resolve it? I am training on the Europarl v7 dataset.

ottokart commented 7 years ago

Are you using the GpuArray backend for Theano? http://deeplearning.net/software/theano/tutorial/using_gpu.html#gpuarray-backend

This Theano issue seems to match your description: https://github.com/Theano/Theano/issues/4101

Laqshay commented 6 years ago

Thanks for your reply. Would Theano 0.9.0 work (Above issue says it is unresolved for 0.8.X releases)? And does it automatically use gpuarray backend if installed, or does it have to be manually configured to do so (Though I couldn't find any instructions on how to configure Theano)?