Fix to ignore tokenizer tests on native for o200k

xebia-functional / xef

Building applications with LLMs through composability, in Kotlin, Scala, ...

https://xef.ai

Apache License 2.0

178 stars 15 forks source link

Fix to ignore tokenizer tests on native for o200k #749

Closed realdavidvega closed 5 months ago

realdavidvega commented 5 months ago

This PR makes the o200k tokenizer tests to be ignored on native, since seems that not all characters are able to be encoded, like the ones from Hindi language. This should happen only on native.

The o200k encoding was added here: https://github.com/xebia-functional/xef/pull/748