Open harveyp123 opened 7 months ago
In short, in the second for loop, for everything minibatch query and passage loss backward, you put the query and passage embedding into the original batch, and calculate the gradient for the current query/passage, so you can make sure the dropout behavior doesn't change your gradient too much.
In our train code, the random states are snapshot using the RandContext class https://github.com/luyug/GC-DPR/blob/79e1fe06bd879662a5e8415efd278080225b9892/train_dense_encoder.py#L53-L69 in the first fwd and restored at the beggining of the 2nd, so what you described shouldn't be a problem.
Oh, okay, I was using deepspeed + gradient caching, the model is wrapped into a deepspeed defined object, and RandContext doesn't work on my side. But it's good to learn from your code : )
The GC-DPR has two steps
However, during the computation, there might be one issues: