yikang-li / MSDN

This is our PyTorch implementation of Multi-level Scene Description Network (MSDN) proposed in our ICCV 2017 paper.
http://cvboy.com/publication/iccv2017_msdn/
227 stars 51 forks source link

GPU memory leakage during evaluation #13

Closed lizhongguo closed 6 years ago

lizhongguo commented 6 years ago

Thanks for your work. I tried to run eval.sh and got an error called "cuda out of memory" . env: -- python 2.7 -- pytorch 0.4.1 -- cuda 9.0 -- gpu nvidia titan x

I found it's caused by a class named "RoIPoolFunction(Function)". In "forward" function of this class , there're some assignment expressions such as "self.output = output" and if I comment these expressions , the code works. I guess , when the model running in the eval mode , tensor like "self.output" wont be released (or grad of tensor ?) and memory leakage happens.