Noticing leaks in release of byte buffers - Githubissues

spring-cloud / spring-cloud-gateway

An API Gateway built on Spring Framework and Spring Boot providing routing and more.

http://cloud.spring.io

Apache License 2.0

4.46k stars 3.28k forks source link

Noticing leaks in release of byte buffers #3399

Open gaurav8628 opened 2 months ago

gaurav8628 commented 2 months ago

Describe the bug We are running an gateway application (has underlying Spring Cloud Gateway v4.0.6) that currently supports Spring Boot 3.1.x. Recently we have started to see issues of memory leaks during the handling of the responses in the filters of SCG that is impacting our application's traffic. We did try to narrow it down by disabling our custom filters and run with default SCG filters and we could still see leaks occurring when SCG is processing the response. (a partial snippet added below).

{"instant":{"epochSecond":1715881307,"nanoOfSecond":613266084},"thread":"egress-h2c-epoll-4","level":"ERROR","loggerName":"io.netty.util.ResourceLeakDetector","message":"LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.\nRecent access records: \n#1:\n\tio.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)\n\tio.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)\n\treactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:296)\n\treactor.netty.channel.FluxReceive.request(FluxReceive.java:131)\n\treactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)\n\treactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:138)\n\treactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)\n\treactor.core.publisher.Operators$BaseFluxToMonoOperator.request(Operators.java:2041)

The above log is printed when we include the jvm argument -Dio.netty.leakDetection.level=paranoid

Springboot Version: 3.1.5 Spring Cloud Gateway Version: 4.0.6 Netty Version: 4.1.100 Jetty Version: 11.0.17

This can be easily reproduced when we have RetryGatewayFilter is enabled.

spencergibb commented 2 months ago

Please try with the latest versions as there have been a number of fixes made in this area

gaurav8628 commented 2 months ago

We tried with 4.0.9, we still see the same issue. We will build a sample app and share it.

spencergibb commented 2 months ago

The latest version is 4.1.3

gaurav8628 commented 1 month ago

We cannot accept 4.1.3 as yet coz as per the dependency of SCG it needs Spring Boot 3.2.x and our applications are not ready with SB 3.2.x as it has many changes. We are currently on SB 3.1.x.

gaurav8628 commented 1 month ago

Hi, I am attaching the applications that I build to reproduce the issues. demoMemoryLeak1 application have the webclient for sending the request running on port 8080 demo2 is the SCG application running on port 8089 stub-app2 is the stub application the on port 8087 Start all three application and send the curl request attached below while true; do curl --http2-prior-knowledge --request GET 'http://localhost:8080/instances' --header 'x-http2-scheme: http' -H "Content-Type: application/json"; done demoMemoryLeak1 2.zip demo 2.zip stub-app 2.zip

gaurav8628 commented 1 month ago

I am also attaching the results of my test run for the simulation of this issue. As per my assessment the issue is arising because the SCG is not able to handle the DATA packet which is having the length as zero. I am attaching the screenshot of tcpdump along with tcpdump file and the logs which shows that Data Leak is happening. Screenshot 2024-05-21 at 12 02 20 PM

gaurav8628 commented 1 month ago

The communication is on http2 please apply this filter to see the traffic for demo2 application. tcp.port==8089 && http2

gaurav8628 commented 1 month ago

Hi Folks, Can you please give a update regarding until when the fix will be delivered for this.

lzolkiewski commented 1 month ago

Hi I'm seeing the same in our logs We are working with spring-cloud-starter-gateway version 4.1.2 @spencergibb could 4.1.3 changes have fixed that? It will be some time before we can upgrade to the latest version I think

io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)
    io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
    reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:298)
    reactor.netty.channel.FluxReceive.request(FluxReceive.java:133)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.request(FluxMap.java:295)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.request(FluxPeekFuseable.java:783)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.request(FluxMap.java:295)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onSubscribe(ChannelSendOperator.java:166)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onSubscribe(FluxMap.java:194)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onSubscribe(FluxPeekFuseable.java:816)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onSubscribe(FluxMap.java:194)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:172)
    reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:150)
    io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Unknown Source)
#2:
    org.springframework.core.io.buffer.NettyDataBufferFactory.wrap(NettyDataBufferFactory.java:94)
    org.springframework.cloud.gateway.filter.NettyWriteResponseFilter.wrap(NettyWriteResponseFilter.java:110)
    org.springframework.cloud.gateway.filter.NettyWriteResponseFilter.lambda$filter$0(NettyWriteResponseFilter.java:87)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:208)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:854)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:294)
    reactor.netty.channel.FluxReceive.request(FluxReceive.java:133)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.request(FluxMap.java:295)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.request(FluxPeekFuseable.java:783)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.request(FluxMap.java:295)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onSubscribe(ChannelSendOperator.java:166)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onSubscribe(FluxMap.java:194)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onSubscribe(FluxPeekFuseable.java:816)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onSubscribe(FluxMap.java:194)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:172)
    reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:150)
    io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Unknown Source)
#3:
    reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:840)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:294)
    reactor.netty.channel.FluxReceive.request(FluxReceive.java:133)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.request(FluxMap.java:295)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.request(FluxPeekFuseable.java:783)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.request(FluxMap.java:295)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.request(FluxContextWriteRestoringThreadLocals.java:163)
    org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onSubscribe(ChannelSendOperator.java:166)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onSubscribe(FluxMap.java:194)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onSubscribe(FluxPeekFuseable.java:816)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxMap$MapConditionalSubscriber.onSubscribe(FluxMap.java:194)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onSubscribe(FluxContextWriteRestoringThreadLocals.java:104)
    reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:172)
    reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:150)
    io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Unknown Source)
#4:
    Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:123)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:30)
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:115)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Unknown Source)
Created at:
    io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:144)
    io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
    io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:108)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:439)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:320)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Unknown Source)

1120562641 commented 1 month ago

I have also encountered such problems, but my personal level is limited and I cannot provide more help. I hope you and people who have encountered such problems can assist the spring team in locating and solving this problem. I feel that from 2.x to 3.x Version problems exist

1120562641 commented 1 month ago

append: gradle dependency id 'org.springframework.boot' version '3.2.6' id 'io.spring.dependency-management' version '1.1.5' org.springframework.cloud:spring-cloud-dependencies:2023.0.2

I tried using the above version, which seems to have fixed the related issues. I still received a leak report, but my production environment memory usage did not continue to increase as before. Maybe you can try the latest version.