Very low BLEU Score during copynet training

tomerwolgithub / Break

MIT License

45 stars 11 forks source link

Very low BLEU Score during copynet training #34

Open justeuer opened 2 years ago

justeuer commented 2 years ago

Hi,

I'm using the default configuration of the copynet model. I updated Allennlp and Pytorch to the latest stable versions (2.7.0, 1.9.1) because I also got the nan error described in this issue. I didn't install the package list given there as this leads to version conflicts. It solved the nan problem, but now the BLEU scores during validation are very low (close to zero). Is that an artifact of the validation loop (the paper doesn't give a BLEU score)?

tomerwolgithub commented 2 years ago

Hi @justeuer. It's been a while, but zero BLEU score should not happen. Note that the exact string match score is a lower-bound on BLEU and it is not zero.