search

reactor / reactor-netty

TCP/HTTP/UDP/QUIC client/server with Reactor over Netty
https://projectreactor.io
Apache License 2.0
2.6k stars 645 forks source link

Spring Cloud API Gateway - Retry is not working due to request body already closed #3500

Open sivarajthava opened 6 days ago

sivarajthava commented 6 days ago

Getting IllegalReferenceCountException when API gateway retrying with internal micro service. The issues started after we upgraded to Spring boot 3.3 (Spring Core 6.1.12) and netty version is 4.1.1.12, the API gateway retry is not working due to below error io.netty.handler.codec.EncoderException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:111) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): *__checkpoint ⇢ org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain] Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83) at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:148)

It is working in Spring boot 2.7 version.

As per initial analysis, data buffer is cleared when we retry 2nd time. So consumer end it is throwing below error due to already stream/buffer closed by client end. org.apache.catalina.connector.ClientAbortException: java.io.EOFException at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:312) at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:615)

Could you please suggest me to fix this issue. Thanks in advance..

violetagg commented 6 days ago

@sivarajthava Please provide a reproducible example

sivarajthava commented 5 days ago

Thanks for your quick response. Step1: Started API gateway and my Employee Micro Service, Now I am posting request (JSON as request body) Step2: Via API gateway the request reached to Employee service and now reading the request body. NOTE: by now, I red the request body Step4: Now I am killing employee micro service Step5: API gateway will retry (retry count is 1) to one more available Employee micro service using name as filter and predicate in spring cloud gateway configuration `filters:

Can we read two times request body in the case of retry? How to achieve Avoid Closing the Request Body for Retry by request body bufferrng?

violetagg commented 5 days ago

@sivarajthava Reactor Netty doesn't provide Retry filter. Reactor Netty has only retry functionality if the connection was closed before sending the request/request body. May be this Retry filter is a functionality provided by the API Gateway but I cannot comment just on the description provided in your comment.