search

lisa-groundhog / GroundHog

Library for implementing RNNs with Theano
BSD 3-Clause "New" or "Revised" License
598 stars 229 forks source link

GPU out of memory problem #30

Closed KHN190 closed 9 years ago

KHN190 commented 9 years ago

I'm running this program with Geforce GTX 750 Ti, where I consistently encounter out of memory problem when the program iterates a specific number of runs.

groundhog.layers.basic: DEBUG: loading W_0_dec_prev_readout_0 of (620, 1000)
GPU status : used 1510.312 Mb free 537.375 Mb, total 2047.688 Mb [context start]
Saving the model...
Model saved.
groundhog.dataset.TM_dataset: DEBUG: 2007723 entries
groundhog.dataset.TM_dataset: DEBUG: Starting from the entry 0
.. iter 0 cost .. # some data
.. iter 5 cost .. # failed on iter 5
Error allocating 74402480 bytes of device memory (out of memory). Driver report 74268672 bytes free and 2147155968 bytes total
# and caused some function error such as attribute is missing

Tried to change state parameters to smaller batch but it doesn't count too much in GPU consuming. Is there a way to control the batches loading to GPU, or if not, how much memory is at least needed?

nouiz commented 9 years ago

Can you update to Theano 0.7? There is a fix for memory usage related to scan you probably use it.

Otherwise, the simplest way is to use a smaller batch size.

On Mon, Mar 30, 2015 at 7:03 AM, enxKSB-K910 notifications@github.com wrote:

I'm running this program with Geforce GTX 750 Ti, where I consistently encounter out of memory problem when the program finishes one run.

groundhog.layers.basic: DEBUG: loading W_0_dec_prev_readout_0 of (620, 1000) GPU status : used 1510.312 Mb free 537.375 Mb, total 2047.688 Mb [context start] Saving the model... Model saved. groundhog.dataset.TM_dataset: DEBUG: 2007723 entries groundhog.dataset.TM_dataset: DEBUG: Starting from the entry 0 .. iter 0 cost .. # some data .. iter 5 cost .. # failed on iter 5 Error allocating 74402480 bytes of device memory (out of memory). Driver report 74268672 bytes free and 2147155968 bytes total

and caused some function error such as attribute is missing

Theano didn't provide an effective way to fix this problem (e.x. how to split batches and holds them when overflows boundary??) and the only working solution seems to be replacing your GPU with one holds larger memoey. Is there a way to control the batches loading to GPU, or if not, how much memory is at least needed?

— Reply to this email directly or view it on GitHub https://github.com/lisa-groundhog/GroundHog/issues/30.

KHN190 commented 9 years ago

I'm using theano 0.7, by checking theano.test(). Also have tried to use smaller batch size such as:

# in state.py
# changed all the values by search
state['bs'] = 10
state['sort_k_batches'] = 5

But in my tests the memory usage didn't drop significantly. Theano.scan is said to minimize GPU transfer, but does not lowered GPU load.

nouiz commented 9 years ago

I do not know how GroundHog dataset work. Do you know if it keep the full dataset on the GPU memory? That could be the cause of problem if the dataset is too big.

On Mon, Mar 30, 2015 at 9:41 AM, enxKSB-K910 notifications@github.com wrote:

I'm using theano 0.7, by checking theano.test(). Also have tried to use smaller batch size such as:

in state.py

changed all the values by search

state['bs'] = 10 state['sort_k_batches'] = 5

But in my tests the memory usage didn't drop significant.

— Reply to this email directly or view it on GitHub https://github.com/lisa-groundhog/GroundHog/issues/30#issuecomment-87685776 .

rizar commented 9 years ago

No we don't do that. Changing the batch size must change memory usage.

What's your version of pytables?

KHN190 commented 9 years ago

@rizar PyTables is 3.1.1, but as @nouiz suggested under theano, I just need to do more gc.collect in GroundHog's mainloop. Though I think there's also a way to change batch size and amount in SGD.py Changes in state.py just result hundred bytes difference, that's not a lot.

And as there has been a solution I should close this issue.