when i train deformable-detr model at 2080Ti(11G) platform, use batch_size=2, it will tip:
RuntimeError: CUDA out of memory. Tried to allocate 56.00 MiB (GPU 0; 10.76 GiB total capacity; 8.54 GiB already allocated; 83.81 MiB free; 9.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF.
and i add 'fp16 = dict(loss_scale='dynamic') 'at the end of configure file, the problem still. so if apex can be added into to improve it?and if can i wan to know how?
when i train deformable-detr model at 2080Ti(11G) platform, use batch_size=2, it will tip:
RuntimeError: CUDA out of memory. Tried to allocate 56.00 MiB (GPU 0; 10.76 GiB total capacity; 8.54 GiB already allocated; 83.81 MiB free; 9.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF.
and i add 'fp16 = dict(loss_scale='dynamic') 'at the end of configure file, the problem still. so if apex can be added into to improve it?and if can i wan to know how?