search

knuddelsgmbh / jtokkit

JTokkit is a Java tokenizer library designed for use with OpenAI models.
https://jtokkit.knuddels.de/
MIT License
516 stars 38 forks source link

added two new embedding model's encoding #85

Closed paplorinc closed 4 months ago

paplorinc commented 5 months ago

See: https://github.com/openai/tiktoken/pull/247

tox-p commented 4 months ago

thanks for the ping, I added them and they will be available in 1.0.0