decreasing bleu score after gan

[jeicy07]

Hi, the accuracy keeps between 0.4-0.6 from epoch 4 to nearly epoch 100 when pretraining the generator. BLEU score on dev set is 19.9 right after pretraining the generator. When training the whole model with gan, BLEU score on dev set is decreasing to 19.7. So I've got some questions: 1) Why does the accuracy of my generator nearly get improved during pretraining? And also from the BLEU score, it seems not good, right? 2) Why the BLEU score decreases instead? Is that because the initial state of generator and discriminator is not so that synchronous, when the discriminator has no idea how to guide the generator? 3) When generating samples, there are nearly 10% words are "UNK", I wonder what you do to these "UNK" words? Thanks

[ZhenYangIACAS]

@jeicy07 I am sorry for my late response. What your translation task? Chinese-English or something else? Additionally, the BLEU score is largely related with the data quality and amount you used? I cannot decide whether your 19.9 BLEU is normal without other information about your data. It is normal for the BLEU decrease at the begin of GAN training. Nearly 10% words are UNK? That is abnormal. Didn't you use BPE?

[wangyirui]

@ZhenYangIACAS Hi, I am training on IWLST14 dataset, I can only make the discriminator achive the acc of 71% and then get overfitted. In joint training, at the beginning, I observed the NLL loss on the validation set keeping (slightly) increasing at the first 6 or 7 epochs. Should we expect the NLL loss on validation set keeping decreasing even at the begin of GAN training? Thanks!

[ZhenYangIACAS]

We observe that the NLL loss will ranges at small scale during the training.