search

PetrochukM / PyTorch-NLP

Basic Utilities for PyTorch Natural Language Processing (NLP)
https://pytorchnlp.readthedocs.io
BSD 3-Clause "New" or "Revised" License
2.21k stars 258 forks source link

bleu metric doesn't seem to work properly #78

Closed HadiZayer closed 5 years ago

HadiZayer commented 5 years ago

I tried to test the get_moses_multi_bleu metric and it doesn't seem to work properly. I ran the following:

x = ['abc']
y = ['abc']
print(get_moses_multi_bleu(x,y))

and it prints 0.0 instead of 1

Edit: I believe the reason is that because the sentence is too short for 2-gram, 3-gram, etc... So there should be a warning message in that case

PetrochukM commented 5 years ago

Please submit this issue to https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl

PyTorch-NLP relies on the official BLEU metric; therefore, by definition, it computes BLEU correctly.