search

ondrejklejch / MT-ComparEval

Tool for comparison and evaluation of machine translation.
Apache License 2.0
56 stars 14 forks source link

Sentence-level BLEU for short sentences #25

Closed martinpopel closed 9 years ago

martinpopel commented 9 years ago

See http://quest.ms.mff.cuni.cz:7280/tasks/212-211/compare

system translation BLEU
Reference Jsem optimista .
commercial1 Já jsem optimista . 0.5946
CU-Chimera Jsem optimista . 0

So CU-Chimera produced a perfect (same-as-reference) translation, but it got sentence-level BLEU=0, while commercial1 produced a different translation and it got non-zero BLEU.

What should be the BLEU of a three-word translation identical to the reference? According to mteval-v13a.pl, it should be 1, because even for n=4, the precision is 1. Why? If the number of test 4-grams is 0, $iscore = 0, but $iscore is used as an exponent, so the 4-gram precision is exp(0).

So I suggest to change this code to $precision = 1 and similarly, in the implementation of recall there should be $recall = 1.

ondrejklejch commented 9 years ago

Done in a39f9c3d815630008657b09b88219a2b78354760.

martinpopel commented 9 years ago

http://quest.ms.mff.cuni.cz:7280/tasks/212-211/compare still shows the BLEU=0 for "Jsem optimista ."

ondrejklejch commented 9 years ago

I'm sorry, I need to reimport tasks.

ondrejklejch commented 9 years ago

I guess it is fixed now: http://quest.ms.mff.cuni.cz:7280/tasks/212-211/compare

martinpopel commented 9 years ago

Yes, it's fixed (just the URL of commercial1 CU-Chimera is now http://quest.ms.mff.cuni.cz:7280/tasks/214-215/compare ).