Batch size for training

[Parskatt]

Am I interpreting correctly that the effective batch size is 4, since it seems you used 4 gpus with batchsize 1 each?

[haoyu94]

yes the batch size is fixed as 1 and 4 gpu is used for training

[Parskatt]

And the learning rate is not multiplied with the number of GPUs, i.e., you use 1e-4, not NUM_GPUS * 1e-4. So if I have one GPU, and batchsize of 1, I should use 1e-4/4 ?

I know the scaling laws for learning rate are not so certain for tiny batchsize, but still.

[haoyu94]

I think you can still try to start from 1e-4. It is a good initial learning rate for many tasks.

As you said, I also think probably the law is not applicable for such a tiny batch size.

[Parskatt]

From some experimenting, I think dividing by 4 is the correct thing to do.