search

amirbar / rnn.wgan

Code for training and evaluation of the model from "Language Generation with Recurrent Generative Adversarial Networks without Pre-training"
https://arxiv.org/abs/1706.01399
252 stars 76 forks source link

RNN with GANs vs Independent RNN Model #8

Closed sunsidazzz closed 7 years ago

sunsidazzz commented 7 years ago

So I am researching in language models that can generate words.

Your results showed that RNN + GANs improves the quality of generated sequences compared to CNN+GANs. But do you think that the combination of RNN and GANs performs better than an independent RNN model?(LSTM, GRU) Because the resulting sentences you got from RNN+GANs are not coherent, and a well-trained RNN model can do the same job. Also, when I was training your model, I feel like using CL+VL+TH is very time-consuming. So is it really worth to train an RNN with GANs? Or the purpose of this project is just to prove that RNN could work well with GANs?

Thanks, Sida Sun

ofirpress commented 7 years ago

The RNN WGAN model is vastly inferior to an RNN trained with the MLE objective. See for example the last page of "Exploring the Limits of Language Modeling" where the results are much better than the ones we obtain. Also, as you state, the training time with MLE is much faster.

This was the first (along with a simultaneous paper published by MILA) work to show that you can train an RNN with just the GAN objective and without MLE. This is just a first step, and we haven't reached the performance of MLE models yet. We believe that if more effort will be put into this research direction we could get to, or even pass, the performance of MLE models with GAN models.

If you have more questions feel free to email any of the authors.

yoosif0 commented 6 years ago

This paper claims that it produces better results than MLE models https://arxiv.org/abs/1801.07736