Closed realdavidvega closed 5 months ago
This PR makes the o200k tokenizer tests to be ignored on native, since seems that not all characters are able to be encoded, like the ones from Hindi language. This should happen only on native.
The o200k encoding was added here: https://github.com/xebia-functional/xef/pull/748
This PR makes the o200k tokenizer tests to be ignored on native, since seems that not all characters are able to be encoded, like the ones from Hindi language. This should happen only on native.
The o200k encoding was added here: https://github.com/xebia-functional/xef/pull/748