search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.3k stars 1.96k forks source link

[BUG] Using stream=true, although the returned data is split, it still waits for the server to complete the processing before returning #35464

Open wzglgo opened 1 year ago

wzglgo commented 1 year ago

I tried to use curl and javasdk to have this effect. If the interface of openAI is called directly, it is asynchronous

azure curl --location --request POST 'https://aicharacter.openai.azure.com/openai/deployments/xxxxx/chat/completions?api-version=2023-03-15-preview' \ --header 'Content-Type: application/json' \ --header 'api-key: xxxxx' \ --data-raw '{ "model": "gpt-3.5-turbo", "messages": [{"role":"user","content":"hello"}], "stream":true }'

openAI curl --location --request POST 'https://api.openai.com/v1/chat/completions' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer sk-xxxxxxx' \ --data-raw '{ "model": "gpt-3.5-turbo", "messages": [ {"role": "user", "content": "hello"} ], "stream": true }'

joshfree commented 1 year ago

Hi @wzglgo thank you for your feedback. @mssfang could you please follow up?

mssfang commented 1 year ago

@wzglgo Thanks for the reach out.

Are you using async client's getChatCompletionsStream?

If not, please check out sample below: Azure ChatCompletions

wzglgo commented 1 year ago

Hello, through testing, I finally found that the stream mode of azure does not respond one by one token, but many tokens respond once, which is different from openai, and according to the effect, it is also better for openai, You can run the two curls I sent earlier to compare and you can see the difference intuitively. Thank you @mssfang

wzglgo commented 1 year ago

This is not a sdk problem, but azure problem, but I don't know what channel to submit to azure

wzglgo commented 1 year ago

@mssfang

mssfang commented 1 year ago

Unfortunately, I don't have direct contact. @trrwilson Do you know?