search

clin1223 / VLDet

[ICLR 2023] PyTorch implementation of VLDet （https://arxiv.org/abs/2211.14843）
Other
177 stars 11 forks source link

Memory Leak problem. #7

Closed muncok closed 1 year ago

muncok commented 1 year ago

Hi,

I am using pytorch==1.10.0 and A100x8 or V100x8 and VLDet_LbaseCCcap_CLIP_R5021k_640b64_2x_ft4x_caption.yaml config.

The GPU memory usage keeps on increasing after each iteration.

increasing speed is about 8~10 MB / iter.

As I figured out, the increment occurs at https://github.com/clin1223/VLDet/blob/c3b9bc52205cd1c48b5b8a43212ee04260f92897/train_net.py#L169

and

Process is finally killed at https://github.com/clin1223/VLDet/blob/c3b9bc52205cd1c48b5b8a43212ee04260f92897/vldet/modeling/roi_heads/vldet_fast_rcnn.py#L422

Do yu have any idea on this?

clin1223 commented 1 year ago

Hi,

Yes, I meet the same problem under the pytorch 1.10.0 CUDA 11.1. Just try pytorch 1.9.0. Thanks!

--Chuang

muncok commented 1 year ago

Hi,

Yes, I meet the same problem under the pytorch 1.10.0 CUDA 11.1. Just try pytorch 1.9.0. Thanks!

--Chuang

After downgrading to 1.9.0, the memory leak disappeared.

Thank you!