Closed Phuoc-Hoan-Le closed 10 months ago
Both use SentencePiece, but these are different models (trained on different datasets, Flores101 and Flores200), see https://github.com/mjpost/sacrebleu/blob/bafca9df5b933770aa5c7fb7d858acd34ac43d4c/sacrebleu/tokenizers/tokenizer_spm.py#L20-L27
Both tokenizers can be used for any dataset and language, but for languages that were not included in the training data of a given tokenizer, we cannot expect reasonable tokenization.
What's the difference between setting "--tokenize" to "flores101" and setting it to "flores200"? Are they the same? They both use SentencePiece tokenizer, right?
If not when is it appropriate to use "flores101" over "flores200" when testing on other datasets beside flores-101/200?