search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.25k stars 1.93k forks source link

[OpenAI] Support token calculation in streaming API #40791

Open mssfang opened 4 days ago

mssfang commented 4 days ago

OpenAI starts to support stream_options in the chat completion API.

We should look into how to support it in Azure OpenAI SDK for Java.

curl payload looks like below, it includes the "usage":{"prompt_tokens":11,"completion_tokens":60,"total_tokens":71}

data: {"id":"xxxW","object":"chat.completion.chunk","created":1719256033,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"I"},"logprobs":null,"finish_reason":null}],"usage":null}

.......

data: {"id":"xxxW","object":"chat.completion.chunk","created":1719256033,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"."},"logprobs":null,"finish_reason":null}],"usage":null}

data: {"id":"xxxW","object":"chat.completion.chunk","created":1719256033,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null}

data: {"id":"xxxW","object":"chat.completion.chunk","created":1719256033,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[],"usage":{"prompt_tokens":11,"completion_tokens":60,"total_tokens":71}}

data: [DONE]