search

geek-ai / Texygen

A text generation benchmarking platform
MIT License
861 stars 202 forks source link

TextGAN and GSGAN generators do not converge #31

Open remidomingues opened 5 years ago

remidomingues commented 5 years ago

Hi, I have been feeding a very simple dataset composed of consecutive ints to all GANs available in this library. The data is generated as follows, and stored in the oracle file.

X = [list(range(20))] * 80

I then attempt to perform a training on real data, using this file in input. Seqgan, Leakgan, Rankgan, Mle and Maligan manage to learn and reproduce this simple pattern.

The generator of GSGAN is able to restrict its vocabulary to the one given in input (i.e. use only integers between 0 and 20 instead of 0 and 5000), but is not able to capture the ordering.

The generator of TextGAN is neither able to restrict its vocabulary to the one given in input, nor able to learn an ordering.

As a result, the nll-test for these models is far from the one reached by the other models. Any hint on what may be causing this issue?

remidomingues commented 5 years ago

Hi again,

As a possible direction, I observed that feeding two identical test samples to the generator of GSGAN resulted in two different losses. This is not the case for SeqGAN, MaliGAN, MLE and RankGAN.