When I tested with coco dataset, it was fine to Evaluation.
But when I do the training, it keeps alerting that:
return forward_call(*input, **kwargs)
File "/home/ubuntu/16T/part4/detrex/detectron2/detectron2/modeling/backbone/swin.py", line 454, in forward
x = blk(x, attn_mask)
File "/home/ubuntu/anaconda3/envs/detrex/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/16T/part4/detrex/detectron2/detectron2/modeling/backbone/swin.py", line 280, in forward
attn_windows = self.attn(x_windows, mask=attn_mask) # nW*B, window_size*window_size, C
File "/home/ubuntu/anaconda3/envs/detrex/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/16T/part4/detrex/detectron2/detectron2/modeling/backbone/swin.py", line 152, in forward
attn = q @ k.transpose(-2, -1)
RuntimeError: CUDA out of memory. Tried to allocate 38.00 MiB (GPU 0; 10.76 GiB total capacity; 9.04 GiB already allocated; 19.94 MiB free; 9.47 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Even if I modify the config file:
# modify dataloader config
dataloader.train.num_workers = 1
# please notice that this is total batch size.
# surpose you're using 4 gpus for training and the batch size for
# each gpu is 16/4 = 4
dataloader.train.total_batch_size = 1
Is there any other place to reduce the memory occupation? Can't 2080Ti even run batch size=1?
My GPU model is 2080Ti 10G
When I tested with coco dataset, it was fine to Evaluation.
But when I do the training, it keeps alerting that:
Even if I modify the config file:
Is there any other place to reduce the memory occupation? Can't 2080Ti even run batch size=1?