Azure openai-api Content transfer token accuracy problem

knuddelsgmbh / jtokkit

JTokkit is a Java tokenizer library designed for use with OpenAI models.

https://jtokkit.knuddels.de/

MIT License

518 stars 38 forks source link

Azure openai-api Content transfer token accuracy problem #28

Closed gitLinan closed 1 year ago

gitLinan commented 1 year ago

I use Azure's OpenAI-api service, the model is GPT-35, there is an error between the result of the token calculation of the question and answer and the result returned by the API service

tox-p commented 1 year ago

Can you provide me with the examples that differed in token count? I would assume that Azure OpenAI uses the same model and encodings under the hood, but I have not used Azure myself

gitLinan commented 1 year ago

I'm sorry it's my problem. When Azure is encapsulated, the input of "prompt" needs to include the JSON calculation of the system's role, which causes inconsistency between the tokens calculated locally and the actual consumption.