search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.33k stars 1.97k forks source link

[FEATURE REQ] Open Ai Add Tool Call Id To Each Element of Chat Completions Stream #38254

Open breckenenneking opened 9 months ago

breckenenneking commented 9 months ago

I noticed when streaming the chat completions utilizing tool calls, the id is only provided in the first element of the stream for the tool call. I would like this to be in each element of the stream since I believe it would help with parallel function calling when multiple tool calls are returned at once.

joshfree commented 9 months ago

@mssfang can you help route this?

jpalvarezl commented 8 months ago

Hello @breckenenneking, thanks for the feedback!

One of the main goals of the services is to try to keep the service behaviour between Azure and non-Azure OpenAI as close to each other as possible. What you suggest, I think, would land as a feature request for the service team. I don't think it would go through though, because of the aforementioned. We wouldn't have any agency to change the behaviour on the non-Azure side of this.

In your particular case, I don't know what your function definitions look like, but in either case, if your functions have arguments, you would still need to aggregate all messages from the stream, to be able to deserialize the arguments. As the updates come in the form of a JSON string in the stream which would only be valid on the last message. (when you finally get finish_reason == TOOL_CALL)

Streaming can be particularly difficult to deal with, specially if you are setting N to a value different than 1 in your request (enabling batching). Then you need to look at index to figure out to which of the simultaneous streams a message corresponds to.

Also, sorry for all the assumptions I've made on your use case. We have this usage sample which is probably too simple but illustrates what I mentioned earlier: usage sample for async tool calls This is also not particularly good if you are trying to launch your function as soon as possible. Since finish_reason will come in the last stream message. But at this point you might be better off not using streaming, I think.

We are happy to hear feedback on how we could make this more "ergonomic". So if you have suggestions on something we could do at the SDK level to improve this, we are happy to have a look. If you can provide a bit more details on how your functions are defined, that would help us understand better how we might be able to improve this.

github-actions[bot] commented 8 months ago

Hi @breckenenneking. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

breckenenneking commented 8 months ago

Something I noticed with the example that you provided is that Line 68 of the GetChatCompletionsToolCallAsyncSample.java file gets the first element of the tool calls list, but I am trying to account for this list potentially returning more than one element in this tool calls list. In this case to me, the id would be needed in each stream element so that if I get maybe 2 elements in the tool calls list, I know what string to append each element to. Would there ever be a case where this tool calls list returns multiple elements? I haven't noticed it returning more than one element in my tests, but I want to be prepared should this arise.

mssfang commented 1 month ago

duplicated issue created issue: https://github.com/Azure/azure-sdk-for-java/issues/40512