Multi-GPU doesn't seem to make much of a difference

[Ardivaba]

Not sure if I'm doing something wrong but training on single A100 vs 8xA100 mpiexec -N 8 doesn't seem to change the training speed even though nvidia-smi is showing all of the gpus in use.

When I compare trained models by both of these setups (identical step size) they don't seem to differ.

Is there some configuration option that I missed?

[Jack000]

hm.. are you looking at the ema checkpoint? at 0.9999 ema it can take more than 10000 steps to see a change

this code implements data parallel training. Each gpu should be doing the same thing just with different data. Larger batch sizes allows for larger learning rates, so you should increase the learning rate to see a change. When training using fp16 the multiplier should be about 20 after warmup. If it's lower than that the learning rate is likely too high, if it's higher it's likely too low.

[Ardivaba]

hm.. are you looking at the ema checkpoint? at 0.9999 ema it can take more than 10000 steps to see a change

this code implements data parallel training. Each gpu should be doing the same thing just with different data. Larger batch sizes allows for larger learning rates, so you should increase the learning rate to see a change. When training using fp16 the multiplier should be about 20 after warmup. If it's lower than that the learning rate is likely too high, if it's higher it's likely too low.

Thank you for the reply, this clarifies things, I'll increase my dataset size and give this another go.

[Ardivaba]

Confirmed that everything works as expected and issue was with the size of my dataset.

[chavinlo]

Confirmed that everything works as expected and issue was with the size of my dataset.

How large was it, just wondering