search

CR-Gjx / LeakGAN

The codes of paper "Long Text Generation via Adversarial Training with Leaked Information" on AAAI 2018. Text generation using GAN and Hierarchical Reinforcement Learning.
https://arxiv.org/abs/1709.08624
576 stars 180 forks source link

I can't get the same results as the paper. #30

Open lrt366 opened 5 years ago

lrt366 commented 5 years ago

I ran the codes which In the path（\LeakGAN-master\Image COCO\Main.py）（LeakGAN-master\Image COCO\eval_bleu.py）. but I can't get the same results as the paper. I just changed the python's version(3.6) and tensorflow-gpu 's version(1.8.0). this is the result i got : BLEU2: 0.902 BLEU3: 0.7837 BLEU4 :0.6206 BLEU5: 0.4208 Please tell me how I can modify it.thanks a lot！

lianyixin commented 5 years ago

I got the same result with you, BLEU2: 0.9027 BLEU3: 0.7619 BLEU4: 0.5890

Hope the author can give us a clear explanation.

lianyixin commented 5 years ago

Besides, it's worth pointing out that the bleu score in the adversarial training just bounces around the final value of pre-train stage, meaning that the adversarial training has no help for enhancing the performance of the generator.

CR-Gjx commented 5 years ago

Thanks for your remind, the decline of results may be caused by the update of NLTK https://github.com/nltk/nltk/pull/1844. The latest NLTK version updates the algorithm of BLEU, which may cause the decline of BLEU. I will rerun the experiment and discuss with you.

lianyixin commented 5 years ago

Thanks for your remind, the decline of results may be caused by the update of NLTK nltk/nltk#1844. The latest NLTK version updates the algorithm of BLEU, which may cause the decline of BLEU. I will rerun the experiment and discuss with you.

Thanks for your quick response. It's very helpful. Looking for your further messages.

bearcatt commented 4 years ago

Hi @CR-Gjx , in Table 3 of your paper, how do you obtain the results of SeqGAN and RankGAN? I can't find any result about image captioning in the paper of SeqGAN. I do find some results about image captioning in RankGAN, but their results are not consistent with those reported in Table 3. Do you reimplement SeqGAN and RankGAN to get those results? If so, what's the detailed architecture of these image captioning models? If not, do the results of SeqGAN and RankGAN also suffer from the BUG in NLTK? I would be really appreciated if you can answer my questions :)

CR-Gjx commented 4 years ago

Hi, @bearcatt , the experiments implemented in our paper is open domain text generation rather than conditional text generation (image caption, translation...). We just use the texts in Coco dataset as open domain datasets. In addition, the update of NLTK also causes the decline of BLEU.