search

zhaozhengChen / ReCAM

The official code of CVPR 2022 paper (Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation).
143 stars 18 forks source link

cuda out of memory #13

Closed YangjiaqiDig closed 2 years ago

YangjiaqiDig commented 2 years ago

Hey, I have another question. When I run the make_sem_seg_labels.py , it is out of memory.

  1. the model was created by train_irn process and I only used 2 GPUs, it works fine. Why load model in make_sem_seg_labels is out of memory even with 4 GPUs?
  2. What devices and memory size you used to train the model? I am using NVIDIA-SMI 4 GPUs and 11175Mb / gpu
zhaozhengChen commented 2 years ago

Yes, it will be out of memory if you use 11GB memory GPU. I use a GPU with 24GB memory. Because we crop the image to 512 during training, but we use the original image size in 'make_sem_seg_labels'. There are some images with large image size.

YangjiaqiDig commented 2 years ago

Yes, it will be out of memory if you use 11GB memory GPU. I use a GPU with 24GB memory. Because we crop the image to 512 during training, but we use the original image size in 'make_sem_seg_labels'. There are some images with large image size.

got it, thanks. does it mean if my own data size is even larger, the 24gb would be possible out of memory as well?

zhaozhengChen commented 2 years ago

Yes, one possible solution is that you can set the scale to a smaller value (e.g. 0.5) in 'make_sem_seg_labels'. But I'm not sure if it will lower the performance.

YangjiaqiDig commented 2 years ago

Yes, one possible solution is that you can set the scale to a smaller value (e.g. 0.5) in 'make_sem_seg_labels'. But I'm not sure if it will lower the performance.

I will try that. btw, do you think it makes sense if I move this part to CPU?

YangjiaqiDig commented 2 years ago

Thanks for your work again. I am closing this issue now and will try a large device or decrease the scale. will get back to you here if meet further issues. Thanks!

zhaozhengChen commented 2 years ago

It should be OK, but I think it may be very slow. 'make_sem_seg_labels' is the slowest part of the whole process.