search

mjpost / sacrebleu

Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
Apache License 2.0
1.03k stars 162 forks source link

BLEU and chrF2 on Chinese #240

Open zhhl9101 opened 11 months ago

zhhl9101 commented 11 months ago

In [1]: from sacrebleu.metrics import BLEU, CHRF ...: refs = [["请关掉灯光。"]] ...: sys = ["请关闭灯光。"] In [2]: bleu = BLEU(trg_lang="zh") In [3]: bleu.corpus_score(sys, refs) Out[3]: BLEU = 37.99 83.3/60.0/25.0/16.7 (BP = 1.000 ratio = 1.000 hyp_len = 6 ref_len = 6) In [4]: bleu.get_signature() Out[4]: nrefs:1|case:mixed|eff:no|tok:zh|smooth:exp|version:2.3.1 In [5]: chrf = CHRF() In [6]: chrf_score = chrf.corpus_score(sys, refs) In [7]: print(chrf_score) Out[7]: chrF2 = 28.06

BLEU score 37.99 and chrF2 score 28.06, these two values ​​do not really reflect the close relationship between refs and sys. What the more appropriate usage on Chinese?

lxz1998 commented 2 months ago

正常人只说"请关下灯"，如果需要在"请关掉灯光。"和"请关闭灯光。"里选一个关掉会比关闭更口语化一点