training phase with faster_rcnn_orpn consume more memory than inference phase

jbwang1997 / OBBDetection

OBBDetection is an oriented object detection library, which is based on MMdetection.

Apache License 2.0

522 stars 112 forks source link

training phase with faster_rcnn_orpn consume more memory than inference phase #95

Closed vpeopleonatank closed 2 years ago

vpeopleonatank commented 2 years ago

I 'm training faster_rcnn_orpn_r50_fpn with samples_per_gpu = 1, It takes up to 7.5GB cuda memory, while inference phase only take about 2GB. Is this behaviour normal, I 've set gpu_assign_thr to 1 but the memory doesn't minimize.

jbwang1997 commented 2 years ago

I haven't paid much attention to memory occupation. I think 7.5 GB cuda memory is durable for me. The cuda memory of torch is actually higher than it used.