Open timostark opened 2 months ago
Same issue for me as well, functions are working but just dirty error logs. can you fix it
issue is reproducible using your own example (StreamingChatSample.java).
@alzimmermsft please follow up on this azure-json error in OpenAI
Thanks for filing this issue @timostark.
@mssfang / @jpalvarezl this appears to be a bug in OpenAIServerSentEvents
and how they're being processed. I see a few possible issues here:
ByteBuffer
will have a backing array (ByteBuffer.array()
), if the ByteBuffer
is native memory this will be false and will result in an exception being thrown.ByteBuffer
s emitted may not cleanly break on a server sent event. I found the cases where the exception is thrown that there is a partial server sent event trailing in the emitted data. Ex, data: {"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":
, which is invalid JSON leading to the exception being seen in this issue. We'll need richer checks to make sure that a fully-formed server sent event is being processed.0xA
and 0xD
to perform checks. While they are correct, many people won't know what these are without researching them (new line characters), please compare against \n
and \r
so someone looking at this understands what's going on right away.Here's a code sample to reproduce this without needing to call Azure OpenAI:
Flux<ByteBuffer> source = Flux.fromIterable(Arrays.asList(
ByteBuffer.wrap("data: {\"choices\":[{\"content_filter_results\":{\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"}".getBytes(StandardCharsets.UTF_8)),
ByteBuffer.wrap(",\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}},\"delta\":{\"content\":\" par\"},\"finish_reason\":null,\"index\":0,\"logprobs\":null}],\"created\":1724446441,\"id\":\"id\",\"model\":\"model\",\"object\":\"chat.completion.chunk\",\"system_fingerprint\":\"fingerprint\"}".getBytes(StandardCharsets.UTF_8))));
OpenAIServerSentEvents parser = new OpenAIServerSentEvents(source, ChatCompletions.class);
parser.getEvents().blockLast();
Forgot to add, when running the reproduction sample, while I didn't see exceptions being thrown anywhere, they are logged (as mentioned by this issue). Add an environment variable of AZURE_LOG_LEVEL=debug
to see these logs happening without needing to add an SLF4J logging implementation.
Any news?
Describe the bug After the change to azure-json in beta 9 there are assertions when using function calling with streaming. Everything is still working, but ofc the console errors are not OK.
Exception or Stack Trace ChatCompletionsToolCall.java public static ChatCompletionsToolCall fromJson(JsonReader jsonReader) throws IOException
--> throws (>20 times)
To Reproduce Steps to reproduce the behavior - Use the code snippet below from the official documentation
Code Snippet
Expected behavior No error messages,,