Closed rahulvigneswaran closed 2 years ago
In fact, we did not consider this point. Usually, the larger batch size or larger image resolution (e.g., CIFAR 32X32, others 224x224) requires more GPU memories (number). Moreover, the learning rate is positively related to the batch size. For example, considering lr=0.025 for batch size 64, then lr=0.05 for batch size 128 or lr=0.1 for batch size 256.
Okay, so you are saying that you changed lr, batch_size, according to the n_gpus
.
Nope, n_gpus does not have a fixed equation with lr and batch_size.
Also, how did you reproduce the results of other papers?
In this work, we reproduced existing methods by running existing GitHub codes. If you want to do experiments, you can try their released config and your own config, and select the better one.
Are you offsetting the
batch_size
for the numbern_gpus
in the config itself?CIFAR100:
n_gpus
= 1batch_size
= 128 Effective batchsize => 128*1 = 128 ?ImageNet-LT:
n_gpus
= 2batch_size
= 64 Effective batchsize => 64*2 = 128 ?iNaturalist18:
n_gpus
= 4batch_size
= 512 Effective batchsize => 512*4 = 2048 ?