search

wusize / ovdet

[CVPR2023] Code Release of Aligning Bag of Regions for Open-Vocabulary Object Detection
https://openaccess.thecvf.com/content/CVPR2023/papers/Wu_Aligning_Bag_of_Regions_for_Open-Vocabulary_Object_Detection_CVPR_2023_paper.pdf
Other
172 stars 4 forks source link

GPU memory increase during training. #17

Closed qsunyuan closed 1 year ago

qsunyuan commented 1 year ago

Excellent work!

I run ov-coco benchmark using the following command on 3090 x 8 with cuda 11.3 and torch1.12.1:

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
bash tools/dist_train.sh configs/baron/ov_coco/baron_kd_faster_rcnn_r50_fpn_syncbn_90kx2.py 8

but the GPU memory keep increase during training. Any suggestion?

wusize commented 1 year ago

No worries about the memory increase as the RPN are trained to recall regions of interest with higher confidence scores. However, half the learning rate and double the training iteration number as you use 8 GPUs instead of 16.