n_gpus vs batch_size

[rahulvigneswaran]

Are you offsetting the batch_size for the number n_gpus in the config itself?

CIFAR100:

• n_gpus = 1
• batch_size = 128 Effective batchsize => 128*1 = 128 ?

ImageNet-LT:

• n_gpus = 2
• batch_size = 64 Effective batchsize => 64*2 = 128 ?

iNaturalist18:

• n_gpus = 4
• batch_size = 512 Effective batchsize => 512*4 = 2048 ?

[Vanint]

In fact, we did not consider this point. Usually, the larger batch size or larger image resolution (e.g., CIFAR 32X32, others 224x224) requires more GPU memories (number). Moreover, the learning rate is positively related to the batch size. For example, considering lr=0.025 for batch size 64, then lr=0.05 for batch size 128 or lr=0.1 for batch size 256.

[rahulvigneswaran]

Okay, so you are saying that you changed lr, batch_size, according to the n_gpus.

[Vanint]

Nope, n_gpus does not have a fixed equation with lr and batch_size.

[rahulvigneswaran]

Also, how did you reproduce the results of other papers?

1. Did you reimplement all the methods or just downloaded their GitHub code?
2. Did you use your own config or used their config?

[Vanint]

In this work, we reproduced existing methods by running existing GitHub codes. If you want to do experiments, you can try their released config and your own config, and select the better one.