search

AIPHES / emnlp19-moverscore

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
MIT License
192 stars 31 forks source link

max length not being used in moverscore.py #9

Closed Alex-Fabbri closed 4 years ago

Alex-Fabbri commented 4 years ago

For pytorch_pretrained_bert==0.6.2, tokenizer.max_len=1000000000000 for BertTokenizer. The max_len works as expected in moverscore_v2.py as it uses the updated transformers repo. Put in a hard-coded len to fix it but just wanted to point it out!

andyweizhao commented 4 years ago

Thanks for the feedback, Alex! The repo is indeed old-fashioned... Besides max-length, I will add two more things soon: normalizing scores (1/1+score) and parallelly-running WMD.

Alex-Fabbri commented 4 years ago

Sounds good, thanks!