search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.25k stars 1.93k forks source link

[BUG] GetChatCompletionsStreamAsyncSample code not working #40710

Open ventaglio opened 2 weeks ago

ventaglio commented 2 weeks ago

Describe the bug I am not able to get to a working asynchronous chat stream example using GPT-4 through Azure SDK for Java. I started from the example code provided here https://github.com/Azure/azure-sdk-for-java/blob/azure-ai-openai_1.0.0-beta.9/sdk/openai/azure-ai-openai/src/samples/java/com/azure/ai/openai/usage/GetChatCompletionsStreamAsyncSample.java but it does not work out of the box. When I executed that code, providing the necessary parameters, I got an error.

Exception or Stack Trace There was an error getting chat completions.java.lang.NullPointerException: The mapper [it.celi.dialog.GetChat$$Lambda$143/0x00000008002a0000] returned a null value.

To Reproduce Just execute the example code provided here: https://github.com/Azure/azure-sdk-for-java/blob/azure-ai-openai_1.0.0-beta.9/sdk/openai/azure-ai-openai/src/samples/java/com/azure/ai/openai/usage/GetChatCompletionsStreamAsyncSample.java

Code Snippet I tried to fix the issue by modifying these lines: if (CoreUtils.isNullOrEmpty(chatCompletions.getChoices())) { return null; } this way if (CoreUtils.isNullOrEmpty(chatCompletions.getChoices())) { return ""; }

This change allowed me to execute the code without NPE errors, but the final behavior is different from what I expected it to be.

Expected behavior I expected to get the response token by token in an asynchronous way, but I instead received only Role = assistant

Then I increased the timeout from 10 to 60 seconds TimeUnit.SECONDS.sleep(60);

This way I got: Role = assistant immediately ... the full response after more or less 15 seconds. So the streaming is not working in an asynchronous way.

Screenshots

Setup (please complete the following information):

Additional context

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

ventaglio commented 2 weeks ago

Update: I managed to get the asynchronous streaming working by using another deployment even if I have not yet understood what is the difference between the two deployments. Is there something I can check to understand why one is not working in asynchronous mode and the other it is, using the same exact Java code?

Anyway the example code contains at least a bug in the return null line.

ventaglio commented 2 weeks ago

Update: we finally understood what was wrong with the first deployment. It had a custom filter that was not configured in "async mode" so it was blocking the asynchronous response. I leave the issue open just to fix the return null line error.