search

visinf / n3net

Neural Nearest Neighbors Networks (NIPS*2018)
Other
283 stars 45 forks source link

Out of Memory at Test Time #12

Open 12dmodel opened 5 years ago

12dmodel commented 5 years ago

Hi,

I'm getting out of memory at test time (16GB on my graphics card). This is true even when I reduce patch size to only 20x20 (x3 channel). However, I was able to train the network at 80x80 and batch size of 32 using only 4.5GB memory.

Do you have any idea what might cause this? I'll look at your code closely in the meantime, but any pointer is greatly appreciated.

I don't have TC installed as I am not using conda. Does TC help you save memory?

Thank you.

12dmodel commented 5 years ago

I was stupid enough not to actually change the input size. However, now I am able to run at 200x200 input size (x 1 batch) at 11GB. It seems like memory consumption doesn't grow linearly with the input size (32x3x80x80 input during training uses only 4.5GB). Is this because there will be more candidate to do NN search, which grows as image_size^2?

tobiasploetz commented 5 years ago

Hi @12dmodel,

I just updated the code with a much more memory efficient implementation of some operations that caused memory bottlenecks so far.

Bests, Tobias