how to set learning rate for single GPU and the performance trained on small batch size

[alphaccw]

I am using a single GPU, so my batch_size ==2

1. Should I use the default setting of learning rate as shown in the following args.lr = lrs[args.dataset.lower()] / 16 * args.batch_size Seems the lr will be very small.

2. what the 16 means in the code above?

3. Have you ever try trained on very small batch_size? For me, after 80 epoches, the default setting for lr, batch_size 2, mIoU is about 0.33. It could be when small batch size, 80 epoches is not enough for good converge. But if you have some experience on single GPU (small batch size), it would be great if we can discuss

Thank you for your code and appriciate if you can help.

[wuhuikai]

1 & 2. The default learning rate is set for batch_size = 16. We use args.lr = lrs[args.dataset.lower()] / 16 * args.batch_size to adjust lr because the larger batch_size is, the larger lr is. Besides, batch_size can be set to 4 for a single GPU (12 G).

3. All my models arre trained with 4 Titan-Xp GPUs. Since the batch_size is small, I suggest that (1) freeze BN, and (2) run multiple iters before backward.

[alphaccw]

@wuhuikai thank you for your quick response and your answers. I will try then.