search

martiansideofthemoon / style-transfer-paraphrase

Official code and data repository for our EMNLP 2020 long paper "Reformulating Unsupervised Style Transfer as Paraphrase Generation" (https://arxiv.org/abs/2010.05700).
http://style.cs.umass.edu
MIT License
228 stars 45 forks source link

Perplexity value during training remains constant #48

Open Yaffa16 opened 2 years ago

Yaffa16 commented 2 years ago

When i train on the Shakespeare data, for both shakespeare_0 or shakespeare_1, the perplexity value in eval_results remins constant perplexity= tensor(135.8836)

Although the model trains till checkpoint=13797 , the quality of trained output is consistently low .

I also tried to run the demoparaphraser script while pointing to the Shakespeare model directory downloaded from the GoogleDrive here, however the quality was still not the same as on the web demo API .

Can you please help with why is this happening

martiansideofthemoon commented 2 years ago

hi @Yaffa16, Thanks for reporting the issue! Is your training loss going down? This is logged in Tensorboard (https://github.com/martiansideofthemoon/style-transfer-paraphrase/blob/master/style_paraphrase/run_lm_finetuning.py#L245) but you could also print it directly.

Also, what's the logging of the evaluation script? (this loop - https://github.com/martiansideofthemoon/style-transfer-paraphrase/blob/master/style_paraphrase/run_lm_finetuning.py#L444)

It should output the perplexity of every checkpoint.