Closed King4819 closed 8 months ago
Hi @King4819, to keep the training process consistent with more gpus, you need to keep the overall batch size the same. You can add the "--update_freq 8" flag to accumulate the gradient of 8 steps.
Hi @King4819, to keep the training process consistent with more gpus, you need to keep the overall batch size the same. You can add the "--update_freq 8" flag to accumulate the gradient of 8 steps.
Excuse me, can I ask more explicitly ?
If your experiment is --batch_size 256 on 8 gpus
I have to keep --batch_size 256, and add "--update_freq 8" ?
Thanks!
Yes, just keep the batch size per gpu num_gpu update_freq the same.
Yes, just keep the batch size per gpu num_gpu update_freq the same.
Excuse me, I want to ask that so the "--batch_size" hypa-parameter specify total batch size ? Or batch size on each gpu ?
For example, when setting --batch_size 256 and --num_gpu 8, each gpu has batch size 256 ? Or each gpu has batch size 256/8 ?
Thanks!
“--bach_size" here is the batch size on each gpu.
Thanks for your reply !
I want to ask that whether the method performs well when training on single gpu ? Are there any hypaparameter settings we need to adjust when training on single gpu ? Thanks!