microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps
https://aka.ms/semantic-kernel
MIT License
21.82k stars 3.25k forks source link

Community Issue: How to customize response handlers? #6220

Closed sophialagerkranspandey closed 2 months ago

sophialagerkranspandey commented 5 months ago

Discussed in https://github.com/microsoft/semantic-kernel/discussions/6198

Originally posted by **niltor** May 12, 2024 ## Problems I encountered When I use `Kernel.InvokePromptStreamingAsync(prompt);`, I have errors: ```log System.ArgumentNullException: Value cannot be null. (Parameter 'value') at Azure.AI.OpenAI.ChatRole.op_Implicit(String value) at Azure.AI.OpenAI.StreamingChatCompletionsUpdate.DeserializeStreamingChatCompletionsUpdates(JsonElement element) at Azure.Core.Sse.SseAsyncEnumerator`1.EnumerateFromSseStream(Stream stream, Func`2 multiElementDeserializer, CancellationToken cancellationToken)+MoveNext() at Azure.Core.Sse.SseAsyncEnumerator`1.EnumerateFromSseStream(Stream stream, Func`2 multiElementDeserializer, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetStreamingChatMessageContentsAsync(ChatHistory chat, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)+MoveNext() at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetStreamingChatMessageContentsAsync(ChatHistory chat, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)+MoveNext() at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetStreamingChatMessageContentsAsync(ChatHistory chat, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() at Microsoft.SemanticKernel.KernelFunctionFromPrompt.InvokeStreamingCoreAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext() at Microsoft.SemanticKernel.KernelFunctionFromPrompt.InvokeStreamingCoreAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext() at Microsoft.SemanticKernel.KernelFunctionFromPrompt.InvokeStreamingCoreAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() at Microsoft.SemanticKernel.KernelFunction.InvokeStreamingAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext() at Microsoft.SemanticKernel.KernelFunction.InvokeStreamingAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+MoveNext() at Microsoft.SemanticKernel.KernelFunction.InvokeStreamingAsync[TResult](Kernel kernel, KernelArguments arguments, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() ``` Because the LLM what I used return data like: ```txt data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":" today","function_call":null,"role":null,"tool_calls":null},"finish_reason":null,"index":0,"logprobs":null}],"created":6030235,"model":"deepseek-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":null} data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":"?","function_call":null,"role":null,"tool_calls":null},"finish_reason":null,"index":0,"logprobs":null}],"created":6030235,"model":"deepseek-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":null} data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":"","function_call":null,"role":null,"tool_calls":null},"finish_reason":"stop","index":0,"logprobs":null}],"created":6030235,"model":"DeepSeek-LLM-67B-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":{"prompt_tokens":16,"completion_tokens":10,"total_tokens":26}} data: [DONE] ``` All the previous data can be obtained. Please pay attention to the end mark (`data: [DONE]`) of the last line. This is not a json format. The parsing may return null and an error may be reported when using it later. ## Solution Is it possible to customize the parsing process so that I can perform special processing on the data returned by different language models and finally process it into a unified format?
github-actions[bot] commented 2 months ago

This issue is stale because it has been open for 90 days with no activity.

markwallace-microsoft commented 2 months ago

@RogerBarret0 is this a duplicate of #8407?

RogerBarret0 commented 2 months ago

@markwallace-microsoft, this is not. The ask can be addressed with @dmytrostruk's answer in the linked discussion.

We also do not support usage of OpenAI Connector against other models content that do not comply if the OpenAI responses standard, relying on OpenAI SDK internal handling of such scenarios.