search

utkuozbulak / pytorch-cnn-visualizations

Pytorch implementation of convolutional neural network visualization techniques
MIT License
7.85k stars 1.49k forks source link

guided_backprop register backward/forward hook seems to consume a lot of GPU memory #53

Closed Terahezi closed 5 years ago

Terahezi commented 5 years ago

Hi, really a great job! I was trying to get a guided-gradcam visualization of a video. But I found as more images have been read, very soon my server shows CUDA out of memory error. Then I discovered that it is because register_backward/forward_hook consumes a lot of GPU memory. Is there any way I could fix this issue? Thanks in advance!

utkuozbulak commented 5 years ago

Hello,

How can it consume a lot of GPU memory? The example implementation in this repository does not even use GPU?

Terahezi commented 5 years ago

Sorry, forgot to mention that. I tried to modify your codes and send both the input and the model to gpu.

Terahezi commented 5 years ago

Hello,

How can it consume a lot of GPU memory? The example implementation in this repository does not even use GPU?

Terahezi commented 5 years ago

Hello, How can it consume a lot of GPU memory? The example implementation in this repository does not even use GPU?

utkuozbulak commented 5 years ago

Pytorch retains the graph of the gradient {edit: not all the time, but under certain circumstances}. If you are not through with scaling up one input to many inputs the retained gradients eventually blow up. I suspect you are having an issue related to this. My suggestion would be to have a look at online resources on this property of Pytorch and what .zero_grad() does.