search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.31k stars 1.96k forks source link

How to know the number of tokens returned when calling the GPT-35 streaming chat interface? #39061

Closed zhuanshenlikai closed 3 months ago

zhuanshenlikai commented 6 months ago

When using the non-streaming interface, I can obtain the number of tokens returned from 'chatCompletions.getUsage().getTotalTokens()'. However, how can I determine the number of tokens returned when using the streaming interface?,the code is below:

image

Is there an API available to query token-related operations? Can you assist me with that? Thank you.

joshfree commented 6 months ago

@mssfang could you please assist @zhuanshenlikai with their question?

mssfang commented 6 months ago

@zhuanshenlikai We don't support token computation for streaming API in OpenAI SDK. You can try out an external library, JTokkit for counting tokens. Also use this website for verify the number of tokens calculated from each streaming message: OpenAI Tokenizer.

For cl100k_base and p50k_base encodings: Java: jtokkit

For r50k_base (gpt2) encodings, tokenizers are available in many languages. Java: gpt2-tokenizer-java

mssfang commented 6 months ago

For more context, OpenAI service doesn't report usage for streaming and we are trying to reach out to service to add this to their response (if possible). Hopefully, we can hear from them back but whether we do extra workaround to add it in SDK or OpenAI getting around to it, we would like to help our customers to have an accurate and consistent behavior. Will keep you posted.

I update the sample on how to use the jtokkit library that might also help for now: https://github.com/Azure/azure-sdk-for-java/blob/62e793bd9b320ec8af3bf96d1a71d9c837dd6204/sdk/openai/azure-ai-openai/src/samples/java/com/azure/ai/openai/StreamingChatSample.java