TER asian support

[esalesky]

I am seeing scores that suggests --ter-asian-support is only applied if --ter-normalize is present. This isn't what I expected from the README, so posting this issue to be safe. This can be verified with wmt21 systems/ref packaged in sacrebleu.
Replicating behavior with sacrebleu version 2.3.1 is below.

Summary:

• With no flags set I get scores X for ja, zh
• With only --ter-asian-support set to true, I get the same scores X (no change)
• With only --ter-normalize set to true, I get scores Y
• With both --ter-normalize and --ter-asian-support set to true, I get scores Z

To replicate with the wmt21 system in the README (shortened output):

sacrebleu -t wmt21/systems -l en-zh --echo NiuTrans | sacrebleu -t wmt21/systems -l en-zh -m ter "name": "TER", "score": 101.3, |norm:no|asian:no|

sacrebleu -t wmt21/systems -l en-zh --echo NiuTrans | sacrebleu -t wmt21/systems -l en-zh -m ter --ter-asian-support "name": "TER", "score": 101.3, |norm:no|asian:yes|

sacrebleu -t wmt21/systems -l en-zh --echo NiuTrans | sacrebleu -t wmt21/systems -l en-zh -m ter --ter-normalize "name": "TER", "score": 101.7, |norm:yes|asian:no|

sacrebleu -t wmt21/systems -l en-zh --echo NiuTrans | sacrebleu -t wmt21/systems -l en-zh -m ter --ter-asian-support --ter-normalize "name": "TER", "score": 54.8, |norm:yes|asian:yes|

[ozancaglayan]

Hi,

Yes it seems that the README is wrong. Looking at the code, normalize is actually what enables tokenization and asian support/tokenization only kicks in if normalize is enabled https://github.com/mjpost/sacrebleu/blob/4f4124642c4eb0b7120e50119c669f0570a326a7/sacrebleu/tokenizers/tokenizer_ter.py#L150