search

crux82 / ganbert-pytorch

Enhancing the BERT training with Semi-supervised Generative Adversarial Networks in Pytorch/HuggingFace
Apache License 2.0
92 stars 17 forks source link

Generator doesn't work #15

Open qlinsey opened 2 years ago

qlinsey commented 2 years ago

Hi, How are you? Thanks for this great work. I have one question: I ran the pytorch code and the generator doesn't work. It means: only model learned and saved, I used generator to generate the random noise and discriminator classify label. However, it always classify to one single or very few labels. I tried so many ways: change noise to normal distribution, add some more layers , but it looks like generator doesn't learn ...

Can please give some guidance or workable code? thanks a lot!

Look forward to hearing from you soon!

crux82 commented 2 years ago

Hi Linsey,

Honestly, I cannot help you understand whether the generator "works" or not. GAN-BERT does not aim to get a good generator. The goal is to constantly challenge the discriminator to improve its quality when few labeled data are available (in a semi-supervised fashion).

We have not thoroughly investigated the quality of the generator-derived material, even considering that it is an MLP applied to random noise. A more in-depth discussion of this topic can be found here:

https://proceedings.neurips.cc/paper/2017/file/79514e888b8f2acacc68738d0cbb803e-Paper.pdf

At the same time, we investigated the possibility to generate real examples but using a different approach here:

https://github.com/crux82/dats

Hope this helps

Bests

Danilo

qlinsey commented 2 years ago

Hi, Danilo,

Thank you so much for your response! It helps!

Linsey