search

mjpost / sacrebleu

Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
Apache License 2.0
1.07k stars 164 forks source link

How to calculate the chrf2 score? #193

Closed RamoramaInteractive closed 2 years ago

RamoramaInteractive commented 2 years ago

I've an output and a reference:

Reference: Das ist das Haus von Tom and Sam Output: Das ist das Haus von Thomas und Samuel

When I use Sacrebleu I got a score of 0.75.

I got as Recall 0.65 ans as Precision 0.5. I used 6-gram as character order and after calculating with the formular on paper I got 0.6132.

Why is my different? My Beta is 2.

ozancaglayan commented 2 years ago

could you maybe check with the original implementation to see if this is a difference that arise in sacreBLEU's chrF implementation?

https://github.com/m-popovic/chrF

ozancaglayan commented 2 years ago

closing as no answers.