search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.32k stars 1.97k forks source link

[BUG] Azure AI Inference API throwing "Unexpected end-of-input: expected close marker for Object" when using ChatCompletionsAsyncClient #42037

Open jdubois opened 1 week ago

jdubois commented 1 week ago

Describe the bug I'm using azure-ai-inference_1.0.0-beta.1.

When I use ChatCompletionsAsyncClient, I got correct answers but there are several log messages that indicates there are errors going underneath.

Here are some of the logs from the sample that I share below:

2024-09-25 15:23:55 [boundedElastic-1] com.azure.core.util.logging.ClientLogger.performLogging()
ERROR: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"delta":{"content":" Java"},"finish_reason":null,"index":0,"logprobs":null}],"created":1727"; line: 1, column: 1]) at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"delta":{"content":" Java"},"finish_reason":null,"index":0,"logprobs":null}],"created":1727"; line: 1, column: 319]
2024-09-25 15:23:55 [boundedElastic-1] com.azure.core.util.logging.ClientLogger.performLogging()
ERROR: com.azure.json.implementation.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"delta":{"content":" Java"},"finish_reason":null,"index":0,"logprobs":null}],"created":1727"; line: 1, column: 1]) at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"delta":{"content":" Java"},"finish_reason":null,"index":0,"logprobs":null}],"created":1727"; line: 1, column: 319]
2024-09-25 15:23:55 [boundedElastic-1] com.azure.core.util.logging.ClientLogger.performLogging()
ERROR: Unexpected end-of-input: was expecting closing '"' for name at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},""; line: 1, column: 180]
2024-09-25 15:23:55 [boundedElastic-1] com.azure.core.util.logging.ClientLogger.performLogging()
ERROR: com.azure.json.implementation.jackson.core.io.JsonEOFException: Unexpected end-of-input: was expecting closing '"' for name at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},""; line: 1, column: 180]
2024-09-25 15:23:55 [boundedElastic-1] com.azure.core.util.logging.ClientLogger.performLogging()
ERROR: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"choices":[{"content_filter_results":{"; line: 1, column: 39]) at [Source: (byte[])"{"choices":[{"content_filter_results":{"; line: 1, column: 40]
2024-09-25 15:23:55 [boundedElastic-1] com.azure.core.util.logging.ClientLogger.performLogging()
ERROR: com.azure.json.implementation.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"choices":[{"content_filter_results":{"; line: 1, column: 39]) at [Source: (byte[])"{"choices":[{"content_filter_results":{"; line: 1, column: 40]
2024-09-25 15:23:55 [boundedElastic-1] com.azure.core.util.logging.ClientLogger.performLogging()
ERROR: Unexpected end-of-input in field name at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"delta":{"content":"less"},"finish_reason":null,"index":0,"logprobs":null}],"created":1727270634,"i"; line: 1, column: 327]
2024-09-25 15:23:55 [boundedElastic-1] com.azure.core.util.logging.ClientLogger.performLogging()
ERROR: com.azure.json.implementation.jackson.core.io.JsonEOFException: Unexpected end-of-input in field name at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"delta":{"content":"less"},"finish_reason":null,"index":0,"logprobs":null}],"created":1727270634,"i"; line: 1, column: 327]

Exception or Stack Trace

There is no stack trace, only the logs shared above.

To Reproduce To reproduce this issue, execute the following code: https://github.com/jdubois/langchain4j-examples/blob/f6932500e8e61abbef15d02e6de8af7e2cfdcb29/github-models-examples/src/main/java/GitHubModelsStreamingChatModelExamples.java#L14

If you want to test the whole example, it's in this branch https://github.com/jdubois/langchain4j-examples/tree/azure-sdk-for-java-issue-2 that I created for this issue.

Code Snippet The code is at https://github.com/jdubois/langchain4j-examples/blob/f6932500e8e61abbef15d02e6de8af7e2cfdcb29/github-models-examples/src/main/java/GitHubModelsStreamingChatModelExamples.java#L14

Expected behavior There should not errors in the logs.

Setup (please complete the following information):

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

alzimmermsft commented 1 week ago

Thanks for filing this issue @jdubois, I believe this is the same issue as https://github.com/Azure/azure-sdk-for-java/issues/41164.

FYI @mssfang, @jpalvarezl

cato-dll commented 1 week ago

I raised a similar and likely related defect in the Langchain4j project - https://github.com/langchain4j/langchain4j/issues/1848,