StreamingChatClient returns empty results with internal issue: MissingRequiredAnnotationException

[radhakrishna67]

StreamingChatClient (azure open ai) returns no response with Spring AI version 0.8.1 ChatClient (azure open ai) returns the response with the same Spring AI version 0.8.1

Bug description StreamingChatClient returns empty results with internal issue: MissingRequiredAnnotationException

Environment Spring Boot 3.2.4 Spring AI version: 0.8.1, Java version: 17.0.2 (Oracle)

Steps to reproduce

Expected behavior A clear and concise description of what you expected to happen.

Minimal Complete Reproducible example `var beanOutputParser = new BeanOutputParser<>(Demo.class);

String promptStr = """ you are an AI assistant to help with xyz products. If not sure, state that don't know. You have to provide {N_QUERIES} recommended queries for generated answer of the given query.

The response should be jackson JSON parsable without any special characters that should have user query, generated answer and {N_QUERIES} recommended queries. Use Bean parser to parse output to the java bean.

PREVIOUS CONVERSATION: {CONVERSATION} {query} {format}

""";

PromptTemplate promptTemplate = new PromptTemplate(promptStr, getReplacements(promptInput, beanOutputParser)); Prompt prompt = promptTemplate.create();

// with chatClient - this works as expected.

Generation generation = chatClient.call(prompt).getResult(); Demo demo = beanOutputParser.parse(generation.getOutput().getContent());

// with streamingChatClient - this returns empty string as response Mono recommendationMono = streamingChatClient.stream(prompt) .map(chatResponse -> { Generation result = chatResponse.getResult(); AssistantMessage output = result.getOutput(); return beanOutputParser.parse(output.getContent()); }) .single();

private static Map<String, Object> getReplacements(PromptInput promptInput, BeanOutputParser<Demo> beanOutputParser) {
    return Map.of("N_QUERIES", promptInput.getNumberOfQueryRecommendations(),
            "query", promptInput.getQuery(),
            "CONVERSATION", "",
            "format", beanOutputParser.getFormat());
}

// Exception when using streamingChatClient com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input at [Source: (String)""; line: 1, column: 0] at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4916) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4818) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3740) at org.springframework.ai.parser.BeanOutputParser.parse(BeanOutputParser.java:113) at com.demo.service.impl.AiRecommenderServiceImpl.lambda$getContextRecommendedQueries$0(AiRecommenderServiceImpl.java:96) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113) at reactor.core.publisher.FluxIterable$IterableSubscription.fastPath(FluxIterable.java:402) at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:291) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) at reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103) at reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2909) at reactor.core.publisher.StrictSubscriber.onSubscribe(StrictSubscriber.java:77) at reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201) at reactor.core.publisher.FluxStream.subscribe(FluxStream.java:69) at reactor.core.publisher.Mono.subscribe(Mono.java:4568) at org.springframework.web.servlet.mvc.method.annotation.ReactiveTypeHandler$DeferredResultSubscriber.connect(ReactiveTypeHandler.java:465) at org.springframework.web.servlet.mvc.method.annotation.ReactiveTypeHandler.handleValue(ReactiveTypeHandler.java:163) at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler.handleReturnValue(ResponseBodyEmitterReturnValueHandler.java:154) at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:136) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:925) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:830) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang.Thread.run(Thread.java:833) `

[tzolov]

@radhakrishna67, can you, please, confirm if this problem is present in current 1.0.0-SNAPSHOT? (0.8.1 is a milestone and the fix with come with 1.0.0-SNAPSHOT and the forthcoming 1.0.0-M1 milestone.)

Also the MissingRequiredAnnotationException is not present in the stack you've shared.

[radhakrishna67]

@tzolov , I am using spring-ai-azure-openai-spring-boot-starter-0.8.1, is a milestone.

MissingRequiredAnnotationException is only seen while debugging as shown in the screenshot.

[radhakrishna67]

@tzolov Just bringing this top of the list for up coming release

[alexcheng1982]

MissingRequiredAnnotationException may not be an issue. I think it happens because IDEA tries to invoke the toString method for debugging purpose.

In your case, you are trying to use BeanOutputParser to parse the JSON output from LLM. In the streaming mode, each ChatResponse only has partial content of the whole JSON data, so the JSON parsing will fail.

[markpollack]

It would be helpful @radhakrishna67 if you can try 1.0 snapshots, but as @alexcheng1982 mentions this maybe expected behavior in this specific environment

[markpollack]

Closing, please reopen if there are any issues with the current code base. Seems like in this case a partial response of json was used to create a new bean resulting in an error.