Closed kinglai closed 2 years ago
8 is a constant factor that corresponds to the number of GPU used in training DPR in the original paper. This scaling here offset the effect of gradient mean allreduce, thereby aligning the gradient value with the original setup.
is 8 just a parameter or has some exact mining? eg, 8 GPUs?