CUDA out of memory, althougt setting batch_size = 2 on 2 gpus

aim-uofa / AdelaiDet

AdelaiDet is an open source toolbox for multiple instance-level detection and recognition tasks.

https://git.io/AdelaiDet

Other

3.38k stars 648 forks source link

CUDA out of memory, althougt setting batch_size = 2 on 2 gpus #452

Open Pxtri2156 opened 3 years ago

Pxtri2156 commented 3 years ago

Hi everyone. I'm trying to finetune ABCnet v2 from CTW1500 on my custom dataset. However, I got an error CUDA of out memory after training some iterations (over 500 iters). Although I'm setting batch size = 2 and using 2 GPUs, I still meet the same error. Can you do me a favour? I wonder whether I can change any config of the model to descrease the size memory of CUDA. Thank you so much for being so attentive.

TAOSHss commented 2 years ago

@Pxtri2156 My memory usage will gradually increase during the training process, resulting in CUDA out of memory；have you solved the problem now