joshfree
commented
5 months ago @mssfang could you please assist @nspyke
Open nspyke opened 5 months ago
joshfree
commented
5 months ago @mssfang could you please assist @nspyke
nspyke
commented
5 months ago Hi @joshfree, thanks for picking this up so quickly.
I have done a bit more investigation, this time using the Azure OpenAI SDK directly and removing the Spring AI library. I've found that the stream does work now, even though the stack trace makes it look like it the issue is coming from deep inside the Azure library.
@Bean
public OpenAIAsyncClient chatClient() {
return new OpenAIClientBuilder()
.credential(new AzureKeyCredential("my-az-open-ai-key"))
.endpoint("https://my-endpoint.openai.azure.com")
.buildAsyncClient();
}And this code in the controller
@PostMapping(path = "stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ChatCompletions> stream(@RequestBody String input) {
List<ChatRequestMessage> chatMessages = new ArrayList<>();
chatMessages.add(new ChatRequestSystemMessage("You are a helpful assistant. You will talk like a pirate."));
chatMessages.add(new ChatRequestUserMessage(input));
return chatClient.getChatCompletionsStream(
"gpt-4",
new ChatCompletionsOptions(chatMessages));
}Back into Spring AI and stepping through with the debugger, it calls
com.azure.ai.openai.implementation.OpenAIClientImpl#getChatCompletionsWithResponse which calls the interface stub
com.azure.ai.openai.implementation.OpenAIClientImpl.OpenAIClientService#getChatCompletionsSync
Response<BinaryData> getChatCompletionsSync(@HostParam("endpoint") String var1, @QueryParam("api-version") String var2, @PathParam("deploymentId") String var3, @HeaderParam("accept") String var4, @BodyParam("application/json") BinaryData var5, RequestOptions var6, Context var7);This looks like a potential cause of the blocking issue.
I would have expected it to call com.azure.ai.openai.implementation.OpenAIClientImpl#getChatCompletionsWithResponseAsync and com.azure.ai.openai.implementation.OpenAIClientImpl.OpenAIClientService#getChatCompletions with the Mono<> return type.
nspyke
commented
5 months ago Duplicated this issue in Spring AI
Describe the bug In a Spring Boot WebFlux project, calling the stream method of Spring AI which then calls the Azure OpenAI SDK, it throws the exception
block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-2Exception or Stack Trace
To Reproduce Create a new Spring Boot 3.3.1 project through initializer, add Spring AI and Azure Open AI dependencies and configure them in application.properties for autoconfiguration.
Code Snippet
Expected behavior Azure OpenAI chat streams a response
Setup (please complete the following information):
Additional context Add any other context about the problem here.
Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report