There are many memory occupation in GPU0

shinya7y / UniverseNet

USB: Universal-Scale Object Detection Benchmark (BMVC 2022)

Apache License 2.0

422 stars 54 forks source link

There are many memory occupation in GPU0 #23

Closed John-Yao closed 3 years ago

John-Yao commented 3 years ago

Hi,

I trained detector in my dataset using 'universenet50_gfl_xxx.py' and there were many memory occupation in GPU0

And I noticed that extra memory occupation occured when the program go to the lines:

model = MMDistributedDataParallel( model.cuda(), device_ids=[torch.cuda.current_device()], broadcast_buffers=False, find_unused_parameters=find_unused_parameters)

Also I try to train detector with cascade rcnn. There are no memory occupation

Could you provide some suggestion?

Thanks!

shinya7y commented 3 years ago

Thank you for pointing out the issue. ImageNet pretrained Res2Net seems to be loaded on GPU0. Related issue: https://github.com/open-mmlab/mmdetection/issues/3646

Please change configs as below:

    pretrained=('https://github.com/shinya7y/weights/releases/download/v1.0.2/'
                'res2net50_v1b_26w_4s-3cf99910_mmdetv2-92ed3313.pth'),

https://github.com/shinya7y/weights/releases/download/v1.0.2/res2net50_v1b_26w_4s-3cf99910_mmdetv2-92ed3313.pth was converted with map_location='cpu'.