surrogate = surrogate * (trainer.distributed_factor / 8.)

luyug / GC-DPR

Train Dense Passage Retriever (DPR) with a single GPU

Other

128 stars 20 forks source link

surrogate = surrogate * (trainer.distributed_factor / 8.) #4

Closed kinglai closed 2 years ago

kinglai commented 2 years ago

is 8 just a parameter or has some exact mining? eg, 8 GPUs?

luyug commented 2 years ago

8 is a constant factor that corresponds to the number of GPU used in training DPR in the original paper. This scaling here offset the effect of gradient mean allreduce, thereby aligning the gradient value with the original setup.