knuddelsgmbh / jtokkit

JTokkit is a Java tokenizer library designed for use with OpenAI models.
https://jtokkit.knuddels.de/
MIT License
518 stars 38 forks source link

May I ask if you support the GPT-3.5-tubor model, and if so, how can I use it? #2

Closed MrWangGang closed 1 year ago

MrWangGang commented 1 year ago

May I ask if you support the GPT-3.5-tubor model, and if so, how can I use it?

tox-p commented 1 year ago

gpt-3.5-turbo uses cl100k_base as encoding, but you can simply use getEncondingForModel:

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding enc = registry.getEncodingForModel(ModelType.GPT_3_5_TURBO);

For further information on how to use Encoding, refer to the docs: https://javadoc.io/doc/com.knuddels/jtokkit/latest/index.html

If you are talking client libraries: this is out of scope for this library, but if you need a convenient API client you can take a lookt at https://github.com/TheoKanning/openai-java

MrWangGang commented 1 year ago

Thank you for your answer. It's very useful to me