search

tamarott / SinGAN

Official pytorch implementation of the paper: "SinGAN: Learning a Generative Model from a Single Natural Image"
https://tamarott.github.io/SinGAN.htm
Other
3.29k stars 608 forks source link

Cuda Out of Memory in gradients_penalty.backward() #141

Closed ilyak93 closed 3 years ago

ilyak93 commented 3 years ago

Hi. I'm playing with the model's architecture a little, and adding even small and not too much memory-expensive additions (like another conv. block and e.t.c), it throws me OutOfMemory exception.

I've noticed there is a lot retain_graph, create_graph which I assume is very costly to the memory. Can you recommend any ways to optimize the memory consumption of the model, because you familiar with all its memory bottlenecks well ?

I couldn't checkpoint the gradients because the use of autograd.grad. Any other suggestions ? Thank you.

prashanth31 commented 3 years ago

@ilyak93 : What did you end up doing to fix this? I have the same issue. In my case, I am unable to train my model.

ilyak93 commented 3 years ago

@ilyak93 : What did you end up doing to fix this? I have the same issue. In my case, I am unable to train my model.

You need a GPU with about 16GB memory capacity. I played with my edited model and adapted it to the heavy memory consumption. That's something needed to be considered when working with this model.