Open zhhl9101 opened 11 months ago
In [1]: from sacrebleu.metrics import BLEU, CHRF ...: refs = [["请关掉灯光。"]] ...: sys = ["请关闭灯光。"] In [2]: bleu = BLEU(trg_lang="zh") In [3]: bleu.corpus_score(sys, refs) Out[3]: BLEU = 37.99 83.3/60.0/25.0/16.7 (BP = 1.000 ratio = 1.000 hyp_len = 6 ref_len = 6) In [4]: bleu.get_signature() Out[4]: nrefs:1|case:mixed|eff:no|tok:zh|smooth:exp|version:2.3.1 In [5]: chrf = CHRF() In [6]: chrf_score = chrf.corpus_score(sys, refs) In [7]: print(chrf_score) Out[7]: chrF2 = 28.06
In [1]: from sacrebleu.metrics import BLEU, CHRF
...: refs = [["请关掉灯光。"]]
...: sys = ["请关闭灯光。"]
In [2]: bleu = BLEU(trg_lang="zh")
In [3]: bleu.corpus_score(sys, refs)
Out[3]: BLEU = 37.99 83.3/60.0/25.0/16.7 (BP = 1.000 ratio = 1.000 hyp_len = 6 ref_len = 6)
In [4]: bleu.get_signature()
Out[4]: nrefs:1|case:mixed|eff:no|tok:zh|smooth:exp|version:2.3.1
In [5]: chrf = CHRF()
In [6]: chrf_score = chrf.corpus_score(sys, refs)
In [7]: print(chrf_score)
Out[7]: chrF2 = 28.06
BLEU score 37.99 and chrF2 score 28.06, these two values do not really reflect the close relationship between refs and sys. What the more appropriate usage on Chinese?
正常人只说"请关下灯",如果需要在"请关掉灯光。"和"请关闭灯光。"里选一个关掉会比关闭更口语化一点
In [1]: from sacrebleu.metrics import BLEU, CHRF
...: refs = [["请关掉灯光。"]]
...: sys = ["请关闭灯光。"]
In [2]: bleu = BLEU(trg_lang="zh")
In [3]: bleu.corpus_score(sys, refs)
Out[3]: BLEU = 37.99 83.3/60.0/25.0/16.7 (BP = 1.000 ratio = 1.000 hyp_len = 6 ref_len = 6)
In [4]: bleu.get_signature()
Out[4]: nrefs:1|case:mixed|eff:no|tok:zh|smooth:exp|version:2.3.1
In [5]: chrf = CHRF()
In [6]: chrf_score = chrf.corpus_score(sys, refs)
In [7]: print(chrf_score)
Out[7]: chrF2 = 28.06
BLEU score 37.99 and chrF2 score 28.06, these two values do not really reflect the close relationship between refs and sys. What the more appropriate usage on Chinese?