I am using your train script to train a deformable detr architecture on a custom dataset.
When I set the total batch size to 1 and use 1 gpu it works.
When I set the total batch size to 2 and use 2 gpus it works.
However, when I use total batch size of 4 and use 4 gpu or set total batch size to 8 and use 8 gpu I receive cuda out of memory error massage, although the batch each gpu receives suppose to have the same memory footprint.
Is there any reasonable reason for this to happen?
Hello, and thanks for the great framework,
I am using your train script to train a deformable detr architecture on a custom dataset. When I set the total batch size to 1 and use 1 gpu it works. When I set the total batch size to 2 and use 2 gpus it works. However, when I use total batch size of 4 and use 4 gpu or set total batch size to 8 and use 8 gpu I receive cuda out of memory error massage, although the batch each gpu receives suppose to have the same memory footprint. Is there any reasonable reason for this to happen?