Closed esalesky closed 1 year ago
Hi,
Yes it seems that the README is wrong. Looking at the code, normalize
is actually what enables tokenization and asian support/tokenization only kicks in if normalize is enabled
https://github.com/mjpost/sacrebleu/blob/4f4124642c4eb0b7120e50119c669f0570a326a7/sacrebleu/tokenizers/tokenizer_ter.py#L150
I am seeing scores that suggests
--ter-asian-support
is only applied if--ter-normalize
is present. This isn't what I expected from the README, so posting this issue to be safe. This can be verified with wmt21 systems/ref packaged in sacrebleu.Replicating behavior with sacrebleu version 2.3.1 is below.
Summary:
--ter-asian-support
set to true, I get the same scores X (no change)--ter-normalize
set to true, I get scores Y--ter-normalize
and--ter-asian-support
set to true, I get scores ZTo replicate with the wmt21 system in the README (shortened output):