search

idiap / fullgrad-saliency

Full-gradient saliency maps
Other
201 stars 31 forks source link

A question about the memory #11

Closed yikekk closed 2 years ago

yikekk commented 3 years ago

May I ask how much memory does this program need when i run this on 500 pictures one by one, the program is killed Memory cgroup out of memory: Kill process 33883 (python) score 19 or sacrifice child

suraj-srinivas commented 2 years ago

Hi,

Sorry for the very late response. I don't see such issues at my end. The memory taken by this program is just that of a forward and backward pass. Could you please provide a minimal example where you see this issue?

yikekk commented 2 years ago

Sorry for the very late response. This problem occurs when using Simple_FullGrad. When I generate an attribution map, I only need a small amount of memory, e.g., 100M. But when I keep generating the attribution map repeatedly, the memory will keep increasing.

suraj-srinivas commented 2 years ago

Ok, it might be an issue with the hooks not being closed. Does this problem not occur with FullGrad or smooth fullgrad?

yikekk commented 2 years ago

Actually, when i keep running dumo_images.py, this error did not happen. There may be some errors in my own code, I will double check it. Thank you very much for your reply.

yikekk commented 2 years ago

Sorry to bother you. But it's weired. Actually, I only changed the part of image loading in dump_image.py. Every time I use Image.open() to read a picture with a specified path, and then I change the dimension of the image to match the sample code: image = image/255.0 image = np.transpose(image,[2,0,1]) image = image [np.newaxis,:, : ,:] image = torch.tensor(image) image = image.to(DEVICE) saliency_method = SimpleFullGrad(model) saliency_map = saliency_method.saliency(image) image = image[0] save_saliency_map(image, saliency_map[0], 'test.png')

suraj-srinivas commented 2 years ago

Hi, so if I understand it correctly, when you change the image loading code from pytorch's imagefolder to PIL's image.open(), you get a memory overflow error?

yikekk commented 2 years ago

Hi, Actually, i'm not sure. When I use gradcam and grad, and the loading code is image.open(), this error does not appear.

suraj-srinivas commented 2 years ago

Could you please send me a minimal dump_images.py like code with the problem so that I can investigate?

yikekk commented 2 years ago

yes, i've sent the email.Thank you very much for your help!

yikekk commented 2 years ago

Hi, i've solved the problem. When I put the definition of saliency_methods on lines 103-104 outside the loop, and commented out lines 55-65, this code worked fine. Thank you very much for your help!

suraj-srinivas commented 2 years ago

Oh yes, when you keep re-initializing saliency_methods a new hook keeps getting created which can cause problems. Glad to know it got solved!