search

SIAnalytics / RHINO

Official implementation of the paper "Hausdorff Distance Matching with Adaptive Query Denoising for Rotated Detection Transformer"
Other
22 stars 0 forks source link

How much cuda memory does dotav15 need? #4

Open kouyuanbo opened 2 months ago

kouyuanbo commented 2 months ago

I try to train rhino with the config 'rhino-4scale_r50_2xb2-12e_dotav15.py' on RTX 4090(24 G)×2,batch size is 2.But the following error message has appeared:

RuntimeError: CUDA out of memory. Tried to allocate 1.03 GiB (GPU 0; 23.65 GiB total capacity; 19.63 GiB already allocated; 735.00 MiB free; 20.85 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
nijkah commented 1 month ago

@kouyuanbo I am sorry for the late response. Can you recheck your settings? In my case, the same setting 'rhino-4scale_r50_2xb2-12e_dotav15.py' with batch_size=2 requires 8300~1570 MB. Note that 'the denoising training' from DINO can result in higher memory usage when there are a lot of GT objects.