reactor / reactor-netty

TCP/HTTP/UDP/QUIC client/server with Reactor over Netty
https://projectreactor.io
Apache License 2.0
2.61k stars 647 forks source link

Intermittent Issue of "Connection has been closed BEFORE response, while sending request body" #3468

Closed dukedom26 closed 1 month ago

dukedom26 commented 1 month ago

We have a few Spring Webflux Application. All of them are using Reactor Netty Core and Reactor Netty Http to create ConnectionProvider and Http Client Respectively to create the WebClient. We are getting intermittently the Connection has been closed BEFORE response, while sending request body error on invoking a call from these services to any downstreams

protected Mono<String> post(String requestPath,
                             String payload,
                             RequestContext context ) {
        return Mono.deferContextual(contextView -> {
            return webClient.build()
                    .post()
                    .uri(uriBuilder -> uriBuilder.path(requestPath).build())
                    .header(AUTHORIZATION, context.get(AUTHORIZATION, String.class))
                    .header(CONTENT_TYPE, APPLICATION_JSON)
                    .bodyValue(payload)
                    .exchangeToMono(clientResponse -> {
                        try (var mdcFrame = LoggingUtils.mdcFrameFromContext(contextView)) {
                            if (clientResponse.statusCode().equals(HttpStatus.OK)) {
                                return clientResponse.bodyToMono(String.class);
                            } else {
                                return Mono.error(new Exception("error"))
                            }
                        }
                    })
                    .retryWhen(Retry.backoff(maxRetry, Duration.ofMillis(waitingTime))
                            .filter(this::isRetryError)
                            .onRetryExhaustedThrow((retryBackoffSpec, retrySignal)
                                    -> new RuntimeException("System error")));
        });
    }

Expected Behavior

The issue shouldn't be happening.

Actual Behavior

The issue is happening quite frequently but not always. Upon retrying, most of the times it works.

2024-10-09 04:48:48.885 [reactor-http-epoll-1] [ERROR] tid=499c2876-e199-45e2-adf3-418848367a3f correlationId= http_request_path=/v1/classify http_query_string=  Exception happened while classification of the document. Exception: org.springframework.web.reactive.function.client.WebClientRequestException: Connection has been closed BEFORE response, while sending request body; nested exception is reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) ~[spring-webflux-5.3.16.jar!/:5.3.16] Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): *__checkpoint ? Request to POST https://org.api.com/document/classify [DefaultWebClient] *__checkpoint ? org.app.filter.CPSWebFilter$$EnhancerBySpringCGLIB$$9a7df50c [DefaultWebFilterChain] *__checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain] *__checkpoint ? HTTP POST "/v1/classify" [ExceptionHandlingWebHandler] Original Stack Trace: at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) ~[spring-webflux-5.3.16.jar!/:5.3.16] at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55) ~[reactor-core-3.4.31.jar!/:3.4.31] at reactor.core.publisher.Mono.subscribe(Mono.java:4490) ~[reactor-core-3.4.31.jar!/:3.4.31] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.31.jar!/:3.4.31] at 
.....
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar!/:4.1.94.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar!/:4.1.94.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.94.Final.jar!/:4.1.94.Final] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body

Your Environment

violetagg commented 1 month ago

@dukedom26 Your Reactor Netty version is not supported anymore. Please update to a supported version (more info here)

violetagg commented 1 month ago

Consider checking https://projectreactor.io/docs/netty/release/reference/index.html#faq.connection-closed

github-actions[bot] commented 1 month ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

github-actions[bot] commented 1 month ago

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.