Cuda out of memory when using multiples gpus

[Asafgendler]

Hello, and thanks for the great framework,

I am using your train script to train a deformable detr architecture on a custom dataset. When I set the total batch size to 1 and use 1 gpu it works. When I set the total batch size to 2 and use 2 gpus it works. However, when I use total batch size of 4 and use 4 gpu or set total batch size to 8 and use 8 gpu I receive cuda out of memory error massage, although the batch each gpu receives suppose to have the same memory footprint. Is there any reasonable reason for this to happen?

[abhishekaich27]

@Asafgendler How did you solve this issue?