search

cshanbo / Smooth_BLEU

Corpus level and sentence level BLEU calculation for machine translation
2 stars 1 forks source link

Maybe the BLEU is wrong #2

Open puzzledTao opened 6 years ago

puzzledTao commented 6 years ago

In the BLEU.py, function modified_precision(references, hypothesis, n):

def modified_precision(references, hypothesis, n):

Extracts all ngrams in hypothesis.

counts = Counter(ngrams(hypothesis, n)) 
if not counts:
    return Fraction(0)
# Extract a union of references' counts.
max_counts = reduce(or_, [Counter(ngrams(ref, n)) for ref in references])
# Assigns the intersection between hypothesis and references' counts.
clipped_counts = {ngram: min(count, max_counts[ngram]) for ngram, count in counts.items()}
return Fraction(sum(clipped_counts.values()), sum(counts.values()))

function Fraction() returns reduction of a fraction. Such as value = Fraction(3, 6), so value.numerator = 1, value.denominator= 2. However, when we statistics the number of n-gram, we should add 3 not 1. Looking forward to your reply.

cshanbo commented 6 years ago

Hmm, thank you for pointing out this, I will check it. Normally we use BLEU from nltk. A PR will be welcomed.