back-propagate the discriminator loss through the generator

richarddwang / electra_pytorch

Pretrain and finetune ELECTRA with fastai and huggingface. (Results of the paper replicated !)

325 stars 42 forks source link

back-propagate the discriminator loss through the generator #8

Closed mehdimashayekhi closed 3 years ago

mehdimashayekhi commented 3 years ago

Hi , Thanks for sharing your code. I have a quick question, in the paper it is mentioned that We don’t back-propagate the discriminator loss through the generator, maybe I have missed it, but where in your code this has been taken care of ? can you refer me to it ? Thanks

richarddwang commented 3 years ago

Hi. There is a sampling between generator and discriminator and sampling is not able to be backward propagated. https://github.com/richarddwang/electra_pytorch/blob/9b2533e62cd1b6126feca323fb7b48480b8c2df0/pretrain.py#L294-L302