Closed BramVanroy closed 1 year ago
Yes it seems that the one in the tokenizer makes more sense. It would be good if you can do a PR and slightly elaborate more in the docstring to reflect what it does more or less? (I'm seeing calls to western and asian normalization methods)
Thanks!
If you would like to contribute to this issue, maybe you can slightly improve your PR to also update the README to handle the following issue? https://github.com/mjpost/sacrebleu/issues/229
In TER, the
normalized
argument has the following docstring:https://github.com/mjpost/sacrebleu/blob/4f4124642c4eb0b7120e50119c669f0570a326a7/sacrebleu/metrics/ter.py#L78
This is passed to the TER tokenizer. However, there it has a different meaning altogether:
https://github.com/mjpost/sacrebleu/blob/4f4124642c4eb0b7120e50119c669f0570a326a7/sacrebleu/tokenizers/tokenizer_ter.py#L129
It seems to me that the description in
TER()
is incorrect. If you can confirm that, I can do a PR to change the docstring.