Closed OrianeN closed 7 months ago
Hi Oriane,
I guess you should create the reference list as the following: [[ref1-segment1, ref1-segment2 ,..], [ref2-segment1, ref2-segment2 ,..], ... ]
For your existing structure, you can use following command to transform it to the desired structure: list(map(list, zip(*refs)))
Hope this helps.
Tomas
Now I get it, I thought the imbrication should be like [[ref1-segment1, ref2-segment1], [ref1-segment2, ref2-segment2], ...]
.
This code solves it then to have the same score as the CLI:
# Compute BLEU with flat refs imbricated once to get [[ref1, ref2, ...]]
bleu = BLEU()
bleu_score_flat = bleu.corpus_score(preds, [refs_flat]).score
print(bleu_score_flat)
print(bleu.get_signature())
0.40243871535386055
nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.4.0
Thank you very much @tofula !
I've tried to compute BLEU scores from the Python API as described in the README https://github.com/mjpost/sacrebleu#using-sacrebleu-from-python, but I get strange results that do not correspond to the score I get from the CLI.
Here's an example to reproduce:
$ sacrebleu -t wmt14 -l en-fr --echo src > wmt14.en-fr.en
$ sacrebleu -t wmt14 -l en-fr --echo src > wmt14.en-fr.fr
My predictions are in wmt14.en-fr.preds.txt (rename it to "wmt14.en-fr.preds")
$ !head {dataset_noext}.preds
When running sacreBLEU in CLI, I get this:
$ sacrebleu {dataset_noext}.fr -i {dataset_noext}.preds -m bleu -w 4
In Python, I first load the references and predictions, both with imbricated references as described in the README, and also with flat references to compare:
BLEU with imbricated refs:
BLEU with flat refs:
The issue seems to be the number of recognized references, which is 3003 when using the Python API compared to the expected 1 with the CLI.
I don't get what I'm doing wrong, can someone help me ?