search

knuddelsgmbh / jtokkit

JTokkit is a Java tokenizer library designed for use with OpenAI models.
https://jtokkit.knuddels.de/
MIT License
554 stars 42 forks source link

support android #39

Closed Trree closed 1 year ago

Trree commented 1 year ago

java.lang.IllegalArgumentException: Unsupported flags: 256

final Pattern regex = Pattern.compile(patternString, Pattern.UNICODE_CHARACTER_CLASS);

tox-p commented 1 year ago

You can use version 0.5.0 for now. There is no difference between 0.5.0 and 0.5.1 on Android because Android always uses the unicode character class for regexes.

In the future I may introduce separate builds to tackle this issue, but currently, Android support is not high on my priority list

tox-p commented 1 year ago

I added a workaround for now, can you verify that 0.6.1 works again on Android?