Question about the use of with no grad

[xiayandi]

Hi Richard,

Thank you so much for the great work of this replication. It helps me a lot in understanding Electra in details.

I have one question regarding your use of "with no_grad" in: https://github.com/richarddwang/electra_pytorch/blob/master/pretrain.py#L294

In my understanding, all the tensors under the scope don't require grads. Is there any specific reason that you use with no_grad here?

Thanks!

[richarddwang]

Hi @xiayandi

1. As explained in the paper, gradient flow is stopped at sampling step between generator and discriminator. More precisely, even without torch.no_grad, gradient computation will be interrupted at argmax in self.sample

2. I use torch.no_grad for two reasons

• We can slightly speed up the operation that add logits and gumbel noise (in self.sample before argmax) by not doing its gradient calculation b/c eventually gradient calculation will be interrupted at argmax.
• To clearly show that the whole sampling step doesn't pass gradients, gradients doesn't flow through two models.

Please tag me if there is any thing else I can help you.