search

PJLab-ADG / 3DTrans

An open-source codebase for exploring autonomous driving pre-training
https://bobrown.github.io/Team_3DTrans.github.io/
Apache License 2.0
585 stars 72 forks source link

Problem about GPU memory #36

Open yux326 opened 4 months ago

yux326 commented 4 months ago

Hello, thanks for you ADA codebase. I try to train pvrcnn with Bi3D, I use kitti as source domain and a custom dataset in kitti format (smaller than kitti) as target domain. A CUDA out of memory problem occured during Stage 2. I use 6 RTX 2080ti (each has 10 GB memory) and set BATCH_SIZE_PER_GPU to 1. The Discriminator training and active evaluating were both done successfully but the CUDA out of memory problem occured after these. Are there any bug in memory management in this code? Or do I need more memory to train?

Looking forward to your response!

JiakangYuan commented 1 month ago

Hello, I guess it is caused by the all_gather operation used in stage 2. I will test on the device with the same GPU memory.