search

reactor / reactor-netty

TCP/HTTP/UDP/QUIC client/server with Reactor over Netty
https://projectreactor.io
Apache License 2.0
2.58k stars 646 forks source link

spring cloud gateway memory leak #2384

Closed yanwenliang closed 1 year ago

yanwenliang commented 2 years ago

Expected behavior No leak

Actual behavior LEAK: ByteBuf.release() was not called before it's garbage-collected

2022-07-14 14:53:03.075 DEBUG   [reactor-http-epoll-3] (Loggers.java:250) - [2f764d98-1, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-07-14 14:53:14.769 DEBUG   [reactor-http-epoll-3] (Loggers.java:250) - [2f764d98-2, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver

2022-07-14 14:57:38.536 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [2f764d98-2, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] Buffered ByteBufHolder in the inbound buffer queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)
#2:
    Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)
Created at:
    io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
    io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
    io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:108)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:359)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:238)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)

Steps to reproduce Unfortunately I have no idea how to reproduce it as this error happens randomly in our service in production.

Spring Cloud Gateway version 3.1.1

Reactor Netty version 1.0.16

JVM version (e.g. java -version) 11.0.14

used -Dio.netty.leakDetection.level=PARANOID -Dio.netty.leakDetection.targetRecords=20 _reactor.netty.channel.LeakDetection:debug reactor.netty.channel.FluxReceive:debug

pderop commented 2 years ago

Hi @yanwenliang ,

Without having a reproducer scenario for the issue, it is impossible to guess just based on the provided logs. From the logs, we see that the application/framework has subscribed to the response data (see "subscribing inbound receiver "logs), but it does not consume it for whatever reasons.

so, can you elaborate more on the issue and try to setup a reproducible sample project ? can you also upgrade to the latest Reactor Netty version 1.0.21 ? and do you have some exceptions in the logs before the problem happens ?

thanks

yanwenliang commented 2 years ago

reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body

reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ HTTP POST "xxxx" [ExceptionHandlingWebHandler]
Original Stack Trace:

2022-07-14 14:53:13.258 ERROR   [reactor-http-epoll-1] (CompositeLog.java:122) - [8ebb749c-2194]  500 Server Error for HTTP POST "xxxx"

reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ HTTP POST "xxxx" [ExceptionHandlingWebHandler]
Original Stack Trace:

reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Direct buffer memory
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na]
    at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na]
    at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[na:na]
    at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:649) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:624) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:203) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:187) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:136) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:126) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:394) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) ~[netty-transport-native-unix-common-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
    at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

2022-07-14 14:53:08.842 ERROR   [reactor-http-epoll-4] (Loggers.java:315) - Operator called default onErrorDropped

reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Direct buffer memory
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na]
    at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na]
    at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[na:na]
    at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:649) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:624) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:203) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:187) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:136) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:126) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:394) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) ~[netty-transport-native-unix-common-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
    at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
yanwenliang commented 2 years ago

can you also upgrade to the latest Reactor Netty version 1.0.21 ?

Ok, I'll try.

violetagg commented 2 years ago

@yanwenliang In addition to the recommendation above, please provide more information how SCG is used. Do you use for example some caching filter etc.

violetagg commented 2 years ago

I've edited your comment to improve the formatting. You might want to check out this Mastering Markdown guide for future reference.

yanwenliang commented 2 years ago

In addition to the recommendation above, please provide more information how SCG is used. Do you use for example some caching filter etc.

@violetagg This is all filters I used.

filters=@ArrayList[
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter@17c2d509}, order = -1]],
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter@36bf84e}, order = 10000]],
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ReactiveLoadBalancerClientFilter@2c58dcb1}, order = 10150]],
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.LoadBalancerServiceInstanceCookieFilter@680a66dd}, order = 10151]],
@OrderedGatewayFilter[[GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter@e042c99}, order = 2147483647]]],
violetagg commented 2 years ago

@yanwenliang Were you able to upgrade? Were you able to prepare a reproducible example?

violetagg commented 2 years ago

@yanwenliang I'm closing this one, we can reopen it when you are able to provide more details.

yanwenliang commented 2 years ago

@yanwenliang Were you able to upgrade? Were you able to prepare a reproducible example?

@violetagg I upgrade reactor-netty to 1.0.21, still have memory leak.

violetagg commented 2 years ago

@yanwenliang Please provide reproducible example.

violetagg commented 2 years ago

@yanwenliang I'm closing this one, we can reopen it when you are able to provide more details.

yanwenliang commented 2 years ago

@violetagg @pderop I find the memory leak reason, it has reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation

I add log at org.springframework.http.server.reactive.AbstractServerHttpResponse#writeAndFlushWith

@Override
public final Mono<Void> writeAndFlushWith(Publisher<? extends Publisher<? extends DataBuffer>> body) {
    return new ChannelSendOperator<>(body, inner -> doCommit(() -> writeAndFlushWithInternal(inner)))
        .doOnError(t -> {
            getHeaders().clearContentHeaders();
            log.info(t);
        });
}

this is the log of full request

2022-09-28 11:08:44.608 DEBUG   [reactor-http-epoll-1] (Loggers.java:247) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] New http connection, requesting read
2022-09-28 11:08:44.608 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-09-28 11:08:44.609 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] REGISTERED
2022-09-28 11:08:44.609 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] ACTIVE
2022-09-28 11:08:45.318 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] READ: 399B
2022-09-28 11:08:45.318 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [09938652, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] Increasing pending responses, now 1
2022-09-28 11:08:45.319 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [09938652-1, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@589ed5b8
2022-09-28 11:08:45.320 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc] Created a new pooled channel, now: 148 active connections, 0 inactive connections and 0 pending acquire requests.
2022-09-28 11:08:45.320 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-09-28 11:08:45.320 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc] REGISTERED
2022-09-28 11:08:45.320 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652-1, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] READ COMPLETE
2022-09-28 11:08:46.904 DEBUG   [reactor-http-epoll-1] (Loggers.java:247) - [43a60ddc] Connecting to [/10.106.191.213:22718].
2022-09-28 11:08:46.904 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc] CONNECT: /10.106.191.213:22718
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:247) - [43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] Registering pool release on close event for channel
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] Channel connected, now: 213 active connections, 2 inactive connections and 0 pending acquire requests.
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] ACTIVE
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}, [connected])
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/, connection=PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}}, [configured])
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] Handler is being applied: {uri=http://10.106.191.213:22718/rest/wd/benchmark/response, method=GET}
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}}, [request_prepared])
2022-09-28 11:08:47.228 TRACE   [reactor-http-epoll-1] (NettyRoutingFilter.java:143) - outbound route: 43a60ddc, inbound: [09938652-600] 
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [09938652-1, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] FluxReceive{pending=0, cancelled=false, inboundDone=true, inboundError=null}: subscribing inbound receiver
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] WRITE: 616B
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] FLUSH
2022-09-28 11:08:47.228 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}}, [request_sent])
2022-09-28 11:08:53.539 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652-1, L:/10.xxxxxx:21218 - R:/10.xxxxxx:47608] READ COMPLETE
2022-09-28 11:08:53.677 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] READ: 2048B
2022-09-28 11:08:53.677 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] Received response (auto-read:false) : [Content-Type=application/json;charset=UTF-8, Date=Wed, 28 Sep 2022 03:08:47 GMT, content-length=617565]
2022-09-28 11:08:53.677 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x43a60ddc, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718]}}, [response_received])
2022-09-28 11:08:53.677 DEBUG   [reactor-http-epoll-1] (Loggers.java:253) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-09-28 11:08:53.678 INFO    [reactor-http-epoll-1] (AbstractServerHttpResponse.java:250) - reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
2022-09-28 11:08:53.681 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] READ: 32768B
2022-09-28 11:08:53.681 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [43a60ddc-1, L:/10.xxxxxx:37286 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-09-28 11:08:57.509 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652-1, L:/10.xxxxxx:21218 ! R:/10.xxxxxx:47608] INACTIVE
2022-09-28 11:08:57.509 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [09938652, L:/10.xxxxxx:21218 ! R:/10.xxxxxx:47608] UNREGISTERED
violetagg commented 2 years ago

@yanwenliang Please provide reproducible example.

yanwenliang commented 2 years ago

@yanwenliang Please provide reproducible example.

@violetagg

https://github.com/yanwenliang/scg-leak-demo

  1. start http-server.py
  2. start spring cloud gateway with -Dio.netty.allocator.type=unpooled -Dio.netty.maxDirectMemory=2560000 -Dio.netty.tryReflectionSetAccessible=true
  3. Use curl localhost:8080/scg/leak debug to pause at reactor.netty.http.HttpOperations#send - if (!channel().isActive()) 
    public NettyOutbound send(Publisher<? extends ByteBuf> source) {
if (!channel().isActive()) {
    return then(Mono.error(AbortedException.beforeSend()));
}
if (source instanceof Mono) {
    return new PostHeadersNettyOutbound(((Mono<ByteBuf>) source)
  4. evaluate expressionchannel().close()
  5. continue the request to end

and you will find the memory will not release.

Thanks!

yanwenliang commented 2 years ago

@violetagg hello, Is there any progress on this issue? Thanks!

pderop commented 2 years ago

@yanwenliang ,

just to tell that I'm now checking your reproducer sample project, thanks.

pderop commented 2 years ago

Hi @yanwenliang,

I did some investigations on the last reproducer you have provided, but I'm confused with it, first because it uses an older reactor-netty version: 1.0.15 but this issue has been initially reported for a 1.0.16 reactor-netty version.

Also, in the initial post of this issue, the following error has been reported, which seems to indicate that the destination server response has been received, is still in the data structure of reactor-netty, but the subscriber (the gateway ?) has not consuming it:

2022-07-14 14:57:38.536 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [2f764d98-2, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] Buffered ByteBufHolder in the inbound buffer queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)

now, the reproducer proposes to add a patch in reactor-netty in order to force all incoming client requests to fail with a "AbortedException: Connection has been closed BEFORE send operation", which is a different use case, and I'm not sure if the patch will (or will not) have a bad side effect (maybe it will cause other problems), and it's a different scenario.

So, sorry to insist but can you please try to reproduce issue without the patch, but with the latest GW version (which is using the reactor-netty 1.0.23 version if I'm correct).

Also, can you enable this logger in TRACE level, it may help to investigate:

org.springframework.cloud.gateway.filter.NettyRoutingFilter

thanks.

pderop commented 2 years ago

(I corrected the previous post, please re read it again from here, not from your mail, thank you)

pderop commented 2 years ago

Some updates, maybe this is a duplicate of #2114. So, in the initial log, nobody seems to consume the buffer for the request Id 2f764d98-2:

2022-07-14 14:57:38.536 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [2f764d98-2, L:/10.xxxxx:xxxx - R:10.xxxxx:xxxx] Buffered ByteBufHolder in the inbound buffer queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)

So can you check in logs if someone is subscribing to the request Id 2f764d98-2 ? Normally you should find a log similar to this:

[2f764d98-2, L:/10.xxxxx:xxxx - R:app.services/10.xxxxx:xxxx FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver

If you don’t find it, this may explain the leak: no one is subscribing to the 2f764d98-2 request id and the buffer remains in the reactor-netty data structure. So then you need to check why the subscription does not happen.

This issue may be a duplicate of #2114 and #2115, where the root cause may come from this exception (check if you find it from logs):

java.lang.IllegalStateException: Only one connection receive subscriber allowed.
    at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:182)

please check #2114, and #2115 for more details.

yanwenliang commented 2 years ago

in log, about request 2f764d98-2, i just find: 

2022-07-14 14:53:14.769 DEBUG   [reactor-http-epoll-3] (Loggers.java:250) - [2f764d98-2, L:/10.xxxxx- R:10.1xxxx/10.xxxx:21784] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver

2022-07-14 14:57:38.536 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)

2022-07-14 14:57:38.536 ERROR   [parallel-2] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)

2022-07-14 14:57:38.544 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)

2022-07-14 14:57:38.560 ERROR   [reactor-http-epoll-4] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)

2022-07-14 14:57:38.574 ERROR   [reactor-http-epoll-2] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [2f764d98-2, L:/10.44.90.55:40228 - R:10.102.125.161/10.102.125.161:21784] Buffered ByteBufHolder in the inbound buffer queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:685)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:829)

and there is no log like:

java.lang.IllegalStateException: Only one connection receive subscriber allowed.
    at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:182)
violetagg commented 2 years ago

@yanwenliang Can you check whether you have the memory leak if you make this change in Spring Cloud Gateway

https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124

if (connection != null && connection.channel().isActive()) {

yanwenliang commented 1 year ago

@yanwenliang Can you check whether you have the memory leak if you make this change in Spring Cloud Gateway

https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124

if (connection != null && connection.channel().isActive()) {

@violetagg Thanks reply! When I change the code, it still has memory leak.

I add log in https://github.com/reactor/reactor-netty/blob/9f839321076e74458343511fa6ade31a0f139408/reactor-netty-http/src/main/java/reactor/netty/http/HttpOperations.java#L115

if (!channel().isActive()) {
        log.info("HttpOperations send is not active {}", channel());
        return then(Mono.error(AbortedException.beforeSend()));
}

And I found this log in my log file:

2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (Loggers.java:271) - HttpOperations send is not active [id: 0xabf65163, L:/10.9xx.xxx.xxx:21428 ! R:/10.2xx.xxx.xxx:50840]

And I add log in https://github.com/spring-projects/spring-framework/blob/5e808ad0183a63944ab08017a13cbc7ed75bc581/spring-web/src/main/java/org/springframework/http/server/reactive/AbstractServerHttpResponse.java#L210

.doOnError(t -> {
    log.info("doOnError: ==============" + t);
    getHeaders().clearContentHeaders();
});

And I also find this log:

2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (AbstractServerHttpResponse.java:247) - doOnError: ==============reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation

The full log is: 

2022-10-22 16:32:53.021 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] New http connection, requesting read
2022-10-22 16:32:53.021 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-10-22 16:32:53.021 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] REGISTERED
2022-10-22 16:32:53.021 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] ACTIVE
2022-10-22 16:33:01.256 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] READ: 395B
2022-10-22 16:33:01.255 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 2048, usedDirectMemory: 2055
2022-10-22 16:33:01.256 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] READ: 395B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 72 65 73 74 2f 77 64 2f 62 65 6e |GET /rest/wd/ben|
|00000010| 63 68 6d 61 72 6b 2f 72 65 73 70 6f 6e 73 65 20 |chmark/response |
|00000020| 48 54 54 50 2f 31 2e 31 0d 0a 70 74 70 2d 66 6c |HTTP/1.1..ptp-fl|
|00000030| 61 67 3a 20 31 0d 0a 6d 69 6e 54 69 6d 65 3a 20 |ag: 1..minTime: |
|00000040| 30 0d 0a 6d 61 78 54 69 6d 65 3a 20 30 0d 0a 74 |0..maxTime: 0..t|
|00000050| 72 61 63 65 2d 63 6f 6e 74 65 78 74 3a 20 7b 22 |race-context: {"|
|00000060| 6c 61 6e 65 49 64 22 3a 22 22 2c 22 73 74 72 65 |laneId":"","stre|
|00000070| 73 73 54 65 73 74 43 6f 6e 74 65 78 74 22 3a 7b |ssTestContext":{|
|00000080| 22 62 69 7a 4e 61 6d 65 22 3a 22 22 2c 22 76 65 |"bizName":"","ve|
|00000090| 72 73 69 6f 6e 22 3a 30 2c 22 74 65 6d 70 6c 61 |rsion":0,"templa|
|000000a0| 74 65 49 64 22 3a 30 2c 22 72 6f 75 74 65 53 63 |teId":0,"routeSc|
|000000b0| 68 65 64 75 6c 65 54 79 70 65 22 3a 30 2c 22 74 |heduleType":0,"t|
|000000c0| 72 61 66 66 69 63 4d 61 72 6b 73 22 3a 7b 22 36 |rafficMarks":{"6|
|000000d0| 22 3a 7b 22 65 6e 74 69 74 69 65 73 22 3a 5b 7b |":{"entities":[{|
|000000e0| 22 6e 61 6d 65 22 3a 22 70 74 70 46 6c 61 67 22 |"name":"ptpFlag"|
|000000f0| 2c 22 65 78 74 72 61 49 6e 66 6f 73 22 3a 7b 22 |,"extraInfos":{"|
|00000100| 70 74 70 54 61 73 6b 22 3a 22 32 39 34 37 36 35 |ptpTask":"294765|
|00000110| 22 7d 7d 5d 7d 7d 7d 7d 0d 0a 6b 74 72 61 63 65 |"}}]}}}}..ktrace|
|00000120| 44 65 62 75 67 49 64 3a 20 39 39 39 39 34 39 35 |DebugId: 9999495|
|00000130| 35 38 31 30 31 33 0d 0a 58 2d 4b 54 72 61 63 65 |581013..X-KTrace|
|00000140| 2d 49 64 2d 45 6e 61 62 6c 65 64 3a 20 31 0d 0a |-Id-Enabled: 1..|
|00000150| 68 6f 73 74 3a 20 31 30 2e 39 34 2e 31 38 38 2e |host: 10.94.188.|
|00000160| 31 37 32 3a 32 31 34 32 38 0d 0a 61 63 63 65 70 |172:21428..accep|
|00000170| 74 3a 20 2a 2f 2a 0d 0a 75 73 65 72 2d 61 67 65 |t: */*..user-age|
|00000180| 6e 74 3a 20 50 54 50 0d 0a 0d 0a                |nt: PTP....     |
+--------+-------------------------------------------------+----------------+
2022-10-22 16:33:01.256 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 2048, usedDirectMemory: 7
2022-10-22 16:33:01.256 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [abf65163, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] Increasing pending responses, now 1
2022-10-22 16:33:01.256 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@5da4b3bf
2022-10-22 16:33:01.257 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f] Created a new pooled channel, now: 0 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-22 16:33:01.257 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-10-22 16:33:01.257 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f] REGISTERED
2022-10-22 16:33:01.257 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] READ COMPLETE
2022-10-22 16:33:02.440 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [5cf2b96f] Connecting to [/10.106.191.213:22718].
2022-10-22 16:33:02.440 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f] CONNECT: /10.106.191.213:22718
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Registering pool release on close event for channel
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Channel connected, now: 35 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] ACTIVE
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}, [connected])
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [configured])
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Handler is being applied: {uri=http://10.106.191.213:22718/rest/wd/benchmark/response, method=GET}
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [request_prepared])
2022-10-22 16:33:04.145 TRACE   [reactor-http-epoll-1] (NettyRoutingFilter.java:143) - outbound route: 5cf2b96f, inbound: [abf65163-19208]
2022-10-22 16:33:04.145 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] FluxReceive{pending=0, cancelled=false, inboundDone=true, inboundError=null}: subscribing inbound receiver
2022-10-22 16:33:04.145 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 256, usedDirectMemory: 263
2022-10-22 16:33:04.145 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 256, usedDirectMemory: 519
2022-10-22 16:33:04.145 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 512, usedDirectMemory: 1031
2022-10-22 16:33:04.150 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] WRITE: 610B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 72 65 73 74 2f 77 64 2f 62 65 6e |GET /rest/wd/ben|
|00000010| 63 68 6d 61 72 6b 2f 72 65 73 70 6f 6e 73 65 20 |chmark/response |
|00000020| 48 54 54 50 2f 31 2e 31 0d 0a 70 74 70 2d 66 6c |HTTP/1.1..ptp-fl|
|00000030| 61 67 3a 20 31 0d 0a 6d 69 6e 54 69 6d 65 3a 20 |ag: 1..minTime: |
|00000040| 30 0d 0a 6d 61 78 54 69 6d 65 3a 20 30 0d 0a 74 |0..maxTime: 0..t|
|00000050| 72 61 63 65 2d 63 6f 6e 74 65 78 74 3a 20 7b 22 |race-context: {"|
|00000060| 6c 61 6e 65 49 64 22 3a 22 22 2c 22 73 74 72 65 |laneId":"","stre|
|00000070| 73 73 54 65 73 74 43 6f 6e 74 65 78 74 22 3a 7b |ssTestContext":{|
|00000080| 22 62 69 7a 4e 61 6d 65 22 3a 22 22 2c 22 76 65 |"bizName":"","ve|
|00000090| 72 73 69 6f 6e 22 3a 30 2c 22 74 65 6d 70 6c 61 |rsion":0,"templa|
|000000a0| 74 65 49 64 22 3a 30 2c 22 72 6f 75 74 65 53 63 |teId":0,"routeSc|
|000000b0| 68 65 64 75 6c 65 54 79 70 65 22 3a 30 2c 22 74 |heduleType":0,"t|
|000000c0| 72 61 66 66 69 63 4d 61 72 6b 73 22 3a 7b 22 36 |rafficMarks":{"6|
|000000d0| 22 3a 7b 22 65 6e 74 69 74 69 65 73 22 3a 5b 7b |":{"entities":[{|
|000000e0| 22 6e 61 6d 65 22 3a 22 70 74 70 46 6c 61 67 22 |"name":"ptpFlag"|
|000000f0| 2c 22 65 78 74 72 61 49 6e 66 6f 73 22 3a 7b 22 |,"extraInfos":{"|
|00000100| 70 74 70 54 61 73 6b 22 3a 22 32 39 34 37 36 35 |ptpTask":"294765|
|00000110| 22 7d 7d 5d 7d 7d 7d 7d 0d 0a 6b 74 72 61 63 65 |"}}]}}}}..ktrace|
|00000120| 44 65 62 75 67 49 64 3a 20 39 39 39 39 34 39 35 |DebugId: 9999495|
|00000130| 35 38 31 30 31 33 0d 0a 58 2d 4b 54 72 61 63 65 |581013..X-KTrace|
|00000140| 2d 49 64 2d 45 6e 61 62 6c 65 64 3a 20 31 0d 0a |-Id-Enabled: 1..|
|00000150| 61 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 75 73 65 |accept: */*..use|
|00000160| 72 2d 61 67 65 6e 74 3a 20 50 54 50 0d 0a 46 6f |r-agent: PTP..Fo|
|00000170| 72 77 61 72 64 65 64 3a 20 70 72 6f 74 6f 3d 68 |rwarded: proto=h|
|00000180| 74 74 70 3b 68 6f 73 74 3d 22 31 30 2e 39 34 2e |ttp;host="10.94.|
|00000190| 31 38 38 2e 31 37 32 3a 32 31 34 32 38 22 3b 66 |188.172:21428";f|
|000001a0| 6f 72 3d 22 31 30 2e 32 38 2e 37 35 2e 31 35 37 |or="10.2xx.xxx.xxx|
|000001b0| 3a 35 30 38 34 30 22 0d 0a 58 2d 46 6f 72 77 61 |:50840"..X-Forwa|
|000001c0| 72 64 65 64 2d 46 6f 72 3a 20 31 30 2e 32 38 2e |rded-For: 10.28.|
|000001d0| 37 35 2e 31 35 37 0d 0a 58 2d 46 6f 72 77 61 72 |75.157..X-Forwar|
|000001e0| 64 65 64 2d 50 72 6f 74 6f 3a 20 68 74 74 70 0d |ded-Proto: http.|
|000001f0| 0a 58 2d 46 6f 72 77 61 72 64 65 64 2d 50 6f 72 |.X-Forwarded-Por|
|00000200| 74 3a 20 32 31 34 32 38 0d 0a 58 2d 46 6f 72 77 |t: 21428..X-Forw|
|00000210| 61 72 64 65 64 2d 48 6f 73 74 3a 20 31 30 2e 39 |arded-Host: 10.9|
|00000220| 34 2e 31 38 38 2e 31 37 32 3a 32 31 34 32 38 0d |4.188.172:21428.|
|00000230| 0a 68 6f 73 74 3a 20 31 30 2e 31 30 36 2e 31 39 |.host: 10.106.19|
|00000240| 31 2e 32 31 33 3a 32 32 37 31 38 0d 0a 63 6f 6e |1.213:22718..con|
|00000250| 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 20 30 0d 0a |tent-length: 0..|
|00000260| 0d 0a                                           |..              |
+--------+-------------------------------------------------+----------------+
2022-10-22 16:33:04.150 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] FLUSH
2022-10-22 16:33:04.151 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 1024, usedDirectMemory: 7
2022-10-22 16:33:04.151 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [request_sent])
2022-10-22 16:33:07.207 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 - R:/10.2xx.xxx.xxx:50840] READ COMPLETE
2022-10-22 16:33:07.834 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 2048, usedDirectMemory: 198663
2022-10-22 16:33:07.840 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ: 2048B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 54 54 50 2f 31 2e 31 20 32 30 30 20 0d 0a 43 |HTTP/1.1 200 ..C|
|00000010| 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 70 |ontent-Type: app|
|00000020| 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 |lication/json;ch|
|00000030| 61 72 73 65 74 3d 55 54 46 2d 38 0d 0a 43 6f 6e |arset=UTF-8..Con|
|00000040| 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 35 34 |tent-Length: 154|
|00000050| 33 38 38 0d 0a 44 61 74 65 3a 20 53 61 74 2c 20 |388..Date: Sat, |
|00000060| 32 32 20 4f 63 74 20 32 30 32 32 20 30 38 3a 33 |22 Oct 2022 08:3|
|00000070| 33 3a 30 33 20 47 4d 54 0d 0a 0d 0a 7b 22 62 6c |3:03 GMT....{"bl|
|00000080| 6f 63 6b 52 65 73 70 6f 6e 73 65 22 3a 7b 22 72 |ockResponse":{"r|
.............................................................................
|000007b0| 69 7a 65 22 3a 36 37 32 36 2c 22 74 79 70 65 22 |ize":6726,"type"|
|000007c0| 3a 30 2c 22 61 6e 69 6d 61 74 65 64 22 3a 66 61 |:0,"animated":fa|
|000007d0| 6c 73 65 2c 22 74 68 75 6d 62 6e 61 69 6c 49 6d |lse,"thumbnailIm|
|000007e0| 61 67 65 22 3a 7b 22 63 64 6e 55 72 6c 73 22 3a |age":{"cdnUrls":|
|000007f0| 5b 7b 22 75 72 6c 22 3a 22 68 74 74 70 73 3a 2f |[{"url":"https:/|
2022-10-22 16:33:07.841 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Received response (auto-read:false) : [Content-Type=application/json;charset=UTF-8, Date=Sat, 22 Oct 2022 08:33:03 GMT, content-length=154388]
2022-10-22 16:33:07.841 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [response_received])
2022-10-22 16:33:07.841 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (Loggers.java:271) - HttpOperations send is not active [id: 0xabf65163, L:/10.9xx.xxx.xxx:21428 ! R:/10.2xx.xxx.xxx:50840]
2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (AbstractServerHttpResponse.java:247) - doOnError: ==============reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 2048, usedDirectMemory: 196615
2022-10-22 16:33:07.841 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 32768, usedDirectMemory: 229383
2022-10-22 16:33:07.936 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ: 32768B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 2f 74 78 2d 66 72 65 65 2d 69 6d 67 73 32 2e 61 |/tx-free-imgs2.a|
|00000010| 63 66 75 6e 2e 63 6e 2f 6b 69 6d 67 2f 62 73 32 |cfun.cn/kimg/bs2|
|00000020| 2f 7a 74 2d 69 6d 61 67 65 2d 68 6f 73 74 2f 43 |/zt-image-host/C|
|00000030| 68 51 77 4f 47 5a 6b 59 54 4a 6c 4e 44 41 79 4d |hQwOGZkYTJlNDAyM|
|00000040| 54 42 68 59 6d 4a 6b 4f 54 51 77 4d 68 43 59 7a |TBhYmJkOTQwMhCYz|
|00000050| 4e 63 76 2e 70 6e 67 22 2c 22 66 72 65 65 54 72 |Ncv.png","freeTr|
|00000060| 61 66 66 69 63 50 72 6f 64 75 63 74 41 62 62 72 |afficProductAbbr|
|00000070| 65 76 69 61 74 69 6f 6e 22 3a 22 22 2c 22 66 72 |eviation":"","fr|
|00000080| 65 65 54 72 61 66 66 69 63 43 64 6e 22 3a 66 61 |eeTrafficCdn":fa|
.............................................................................
|00007f90| 69 6d 67 73 2e 61 69 78 69 66 61 6e 2e 63 6f 6d |imgs.aixifan.com|
|00007fa0| 2f 73 74 79 6c 65 2f 69 6d 61 67 65 2f 32 30 31 |/style/image/201|
|00007fb0| 39 30 37 2f 70 4f 6c 4d 4c 45 66 64 6a 53 31 70 |907/pOlMLEfdjS1p|
|00007fc0| 49 50 69 58 37 74 43 59 6b 56 64 41 69 79 47 56 |IPiX7tCYkVdAiyGV|
|00007fd0| 30 48 79 48 2e 6a 70 67 22 2c 22 76 65 72 69 66 |0HyH.jpg","verif|
|00007fe0| 69 65 64 22 3a 30 2c 22 76 69 64 65 6f 49 64 22 |ied":0,"videoId"|
|00007ff0| 3a 31 30 37 31 33 35 39 38 2c 22 73 75 62 54 69 |:10713598,"subTi|
+--------+-------------------------------------------------+----------------+
2022-10-22 16:33:07.936 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-22 16:33:24.833 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163-1, L:/10.9xx.xxx.xxx:21428 ! R:/10.2xx.xxx.xxx:50840] INACTIVE
2022-10-22 16:33:24.833 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [abf65163, L:/10.9xx.xxx.xxx:21428 ! R:/10.2xx.xxx.xxx:50840] UNREGISTERED
2022-10-22 16:33:25.663 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 65536, usedDirectMemory: 3014663
2022-10-22 16:33:25.860 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ: 65536B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 74 6c 65 22 3a 22 22 2c 22 6c 69 6e 6b 22 3a 22 |tle":"","link":"|
|00000010| 2f 76 2f 61 63 31 30 39 31 34 36 35 39 22 2c 22 |/v/ac10914659","|
|00000020| 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 e5 8a |description":"..|
|00000030| a9 e6 89 8b ef bc 9a e8 80 81 e6 9d bf ef bc 8c |................|
|00000040| e3 80 8a e9 a3 8e e4 ba 91 e3 80 8b e5 b7 b2 e7 |................|
|00000050| bb 8f e6 88 90 e5 8a 9f e5 8d 87 e7 ba a7 e4 b8 |................|
|00000060| ba e6 a0 bc e6 96 97 e6 b8 b8 e6 88 8f ef bc 8c |................|
|00000070| e7 ac ac e4 b8 80 e8 bd ae e5 b0 8f 42 4f 53 53 |............BOSS|
.............................................................................
|0000ff90| 69 6e 6b 22 3a 22 68 74 74 70 3a 2f 2f 77 77 77 |ink":"http://www|
|0000ffa0| 2e 61 63 66 75 6e 2e 63 6e 2f 73 65 61 72 63 68 |.acfun.cn/search|
|0000ffb0| 2f 3f 23 70 61 67 65 3d 31 3b 71 75 65 72 79 3d |/?#page=1;query=|
|0000ffc0| 25 45 39 25 38 37 25 39 31 25 45 35 25 38 35 25 |%E9%87%91%E5%85%|
|0000ffd0| 38 39 25 45 35 25 42 45 25 41 31 25 45 34 25 42 |89%E5%BE%A1%E4%B|
|0000ffe0| 39 25 39 44 25 45 37 25 39 35 25 38 43 3b 74 79 |9%9D%E7%95%8C;ty|
|0000fff0| 70 65 3d 76 69 64 65 6f 22 2c 22 64 65 73 63 72 |pe=video","descr|
+--------+-------------------------------------------------+----------------+
2022-10-22 16:33:25.860 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:100) - [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] EXCEPTION: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 65536 byte(s) of direct memory (used: 8323079, max: 8388608)
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 65536 byte(s) of direct memory (used: 8323079, max: 8388608)
        at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:807) ~[classes!/:4.1.75.Final]
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:736) ~[classes!/:4.1.75.Final]
        at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:186) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledByteBufAllocator.java:181) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:91) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) ~[netty-transport-native-unix-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [response_completed])
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718]}}, [disconnecting])
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] CLOSE
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] Channel closed, now: 579 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-22 16:33:56.339 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] USER_EVENT: io.netty.channel.socket.ChannelInputShutdownReadComplete@212f6dea
2022-10-22 16:33:56.662 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] INACTIVE
2022-10-22 16:33:56.662 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0x5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718]}, [disconnecting])
2022-10-22 16:33:56.662 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [5cf2b96f, L:/10.9xx.xxx.xxx:46636 ! R:10.106.191.213/10.106.191.213:22718] UNREGISTERED
2022-10-22 16:33:59.533 ERROR   [reactor-http-epoll-1] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        Hint: [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Buffered ByteBufHolder in the inbound buffer queue
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
        reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
        reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1567)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1561)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#4:
        Hint: 'reactor.left.loggingHandler' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#5:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
Created at:
        io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
: 13 leak records were discarded because they were duplicates
: 65534 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.
2022-10-22 16:33:59.583 ERROR   [reactor-http-epoll-1] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        Hint: [5cf2b96f-1, L:/10.9xx.xxx.xxx:46636 - R:10.106.191.213/10.106.191.213:22718] Buffered ByteBufHolder in the inbound buffer queue
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
        reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
        reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1567)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1561)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#4:
        Hint: 'reactor.left.loggingHandler' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#5:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
Created at:
        io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
: 14 leak records were discarded because they were duplicates
: 131081 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.
violetagg commented 1 year ago

@yanwenliang You closed the issue?

yanwenliang commented 1 year ago

@yanwenliang You closed the issue?

sorry，It was a mistake.

yanwenliang commented 1 year ago

@violetagg I reopen it, and please see the full log of above, Thanks.

violetagg commented 1 year ago

@yanwenliang Yes this time the issue is different. I'm going to prepare one PR and will ask you to test it.

violetagg commented 1 year ago

@yanwenliang Actually can you try the same file https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124

if (connection != null) {

yanwenliang commented 1 year ago

@yanwenliang Actually can you try the same file https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124

if (connection != null) {

@violetagg When I change the code above, the memory leak disappeared.

And the log I added by myself in https://github.com/spring-projects/spring-framework/blob/5e808ad0183a63944ab08017a13cbc7ed75bc581/spring-web/src/main/java/org/springframework/http/server/reactive/AbstractServerHttpResponse.java#L210 is also disappeared.

.doOnError(t -> {
    log.info("doOnError: ==============" + t);
    getHeaders().clearContentHeaders();
});

I can't understand

violetagg commented 1 year ago

@yanwenliang Can you add logs here

https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L102

https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L69

yanwenliang commented 1 year ago

@yanwenliang Actually can you try the same file https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L124 if (connection != null) {

@violetagg When I change the code above, the memory leak disappeared.

And the log I added by myself in https://github.com/spring-projects/spring-framework/blob/5e808ad0183a63944ab08017a13cbc7ed75bc581/spring-web/src/main/java/org/springframework/http/server/reactive/AbstractServerHttpResponse.java#L210 is also disappeared.

.doOnError(t -> {
    log.info("doOnError: ==============" + t);
    getHeaders().clearContentHeaders();
});

I can't understand

@violetagg Unfortunately, the memory leak appear again

violetagg commented 1 year ago

@yanwenliang Please provide the logs

yanwenliang commented 1 year ago

@yanwenliang Please provide the logs

@violetagg 

2022-10-25 16:25:29.395 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [d4f290af, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] New http connection, requesting read
2022-10-25 16:25:29.395 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [d4f290af, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-10-25 16:25:29.395 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [d4f290af, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] REGISTERED
2022-10-25 16:25:29.395 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [d4f290af, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] ACTIVE
2022-10-25 16:25:30.042 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [d4f290af, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] READ: 394B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 72 65 73 74 2f 77 64 2f 62 65 6e |GET /rest/wd/ben|
|00000010| 63 68 6d 61 72 6b 2f 72 65 73 70 6f 6e 73 65 20 |chmark/response |
|00000020| 48 54 54 50 2f 31 2e 31 0d 0a 70 74 70 2d 66 6c |HTTP/1.1..ptp-fl|
|00000030| 61 67 3a 20 31 0d 0a 6d 69 6e 54 69 6d 65 3a 20 |ag: 1..minTime: |
|00000040| 30 0d 0a 6d 61 78 54 69 6d 65 3a 20 30 0d 0a 74 |0..maxTime: 0..t|
|00000050| 72 61 63 65 2d 63 6f 6e 74 65 78 74 3a 20 7b 22 |race-context: {"|
|00000060| 6c 61 6e 65 49 64 22 3a 22 22 2c 22 73 74 72 65 |laneId":"","stre|
|00000070| 73 73 54 65 73 74 43 6f 6e 74 65 78 74 22 3a 7b |ssTestContext":{|
|00000080| 22 62 69 7a 4e 61 6d 65 22 3a 22 22 2c 22 76 65 |"bizName":"","ve|
|00000090| 72 73 69 6f 6e 22 3a 30 2c 22 74 65 6d 70 6c 61 |rsion":0,"templa|
|000000a0| 74 65 49 64 22 3a 30 2c 22 72 6f 75 74 65 53 63 |teId":0,"routeSc|
|000000b0| 68 65 64 75 6c 65 54 79 70 65 22 3a 30 2c 22 74 |heduleType":0,"t|
|000000c0| 72 61 66 66 69 63 4d 61 72 6b 73 22 3a 7b 22 36 |rafficMarks":{"6|
|000000d0| 22 3a 7b 22 65 6e 74 69 74 69 65 73 22 3a 5b 7b |":{"entities":[{|
|000000e0| 22 6e 61 6d 65 22 3a 22 70 74 70 46 6c 61 67 22 |"name":"ptpFlag"|
|000000f0| 2c 22 65 78 74 72 61 49 6e 66 6f 73 22 3a 7b 22 |,"extraInfos":{"|
|00000100| 70 74 70 54 61 73 6b 22 3a 22 32 39 35 39 38 38 |ptpTask":"295988|
|00000110| 22 7d 7d 5d 7d 7d 7d 7d 0d 0a 6b 74 72 61 63 65 |"}}]}}}}..ktrace|
|00000120| 44 65 62 75 67 49 64 3a 20 39 39 39 39 32 35 39 |DebugId: 9999259|
|00000130| 36 32 36 33 35 36 0d 0a 58 2d 4b 54 72 61 63 65 |626356..X-KTrace|
|00000140| 2d 49 64 2d 45 6e 61 62 6c 65 64 3a 20 31 0d 0a |-Id-Enabled: 1..|
|00000150| 68 6f 73 74 3a 20 31 30 2e 39 36 2e 37 30 2e 31 |host: 10.96.70.1|
|00000160| 36 32 3a 32 31 32 34 34 0d 0a 61 63 63 65 70 74 |62:21244..accept|
|00000170| 3a 20 2a 2f 2a 0d 0a 75 73 65 72 2d 61 67 65 6e |: */*..user-agen|
|00000180| 74 3a 20 50 54 50 0d 0a 0d 0a                   |t: PTP....      |
+--------+-------------------------------------------------+----------------+
2022-10-25 16:25:30.042 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 2048, all: 7
2022-10-25 16:25:30.042 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [d4f290af, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] Increasing pending responses, now 1
2022-10-25 16:25:30.042 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [d4f290af-1, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@e0d02d6
2022-10-25 16:25:30.043 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00] Created a new pooled channel, now: 0 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-25 16:25:30.043 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-10-25 16:25:30.043 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00] REGISTERED
2022-10-25 16:25:30.043 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [d4f290af-1, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] READ COMPLETE
2022-10-25 16:25:31.997 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [ac4f0d00] Connecting to [/10.106.191.213:22718].
2022-10-25 16:25:31.997 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00] CONNECT: /10.106.191.213:22718
2022-10-25 16:25:33.074 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Registering pool release on close event for channel
2022-10-25 16:25:33.074 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Channel connected, now: 111 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-25 16:25:33.074 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] ACTIVE
2022-10-25 16:25:33.074 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0xac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718]}, [connected])
2022-10-25 16:25:33.074 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/, connection=PooledConnection{channel=[id: 0xac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718]}}, [configured])
2022-10-25 16:25:33.074 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Handler is being applied: {uri=http://10.106.191.213:22718/rest/wd/benchmark/response, method=GET}
2022-10-25 16:25:33.074 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0xac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718]}}, [request_prepared])
2022-10-25 16:25:33.074 TRACE   [reactor-http-epoll-1] (NettyRoutingFilter.java:143) - outbound route: ac4f0d00, inbound: [d4f290af-2863]
2022-10-25 16:25:33.074 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [d4f290af-1, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] FluxReceive{pending=0, cancelled=false, inboundDone=true, inboundError=null}: subscribing inbound receiver
2022-10-25 16:25:33.074 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 256, all: 263
2022-10-25 16:25:33.074 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 256, all: 519
2022-10-25 16:25:33.075 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 512, all: 1031
2022-10-25 16:25:33.082 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] WRITE: 608B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
|00000000| 47 45 54 20 2f 72 65 73 74 2f 77 64 2f 62 65 6e |GET /rest/wd/ben|
|00000010| 63 68 6d 61 72 6b 2f 72 65 73 70 6f 6e 73 65 20 |chmark/response |
|00000020| 48 54 54 50 2f 31 2e 31 0d 0a 70 74 70 2d 66 6c |HTTP/1.1..ptp-fl|
|00000030| 61 67 3a 20 31 0d 0a 6d 69 6e 54 69 6d 65 3a 20 |ag: 1..minTime: |
|00000040| 30 0d 0a 6d 61 78 54 69 6d 65 3a 20 30 0d 0a 74 |0..maxTime: 0..t|
|00000050| 72 61 63 65 2d 63 6f 6e 74 65 78 74 3a 20 7b 22 |race-context: {"|
|00000060| 6c 61 6e 65 49 64 22 3a 22 22 2c 22 73 74 72 65 |laneId":"","stre|
|00000070| 73 73 54 65 73 74 43 6f 6e 74 65 78 74 22 3a 7b |ssTestContext":{|
|00000080| 22 62 69 7a 4e 61 6d 65 22 3a 22 22 2c 22 76 65 |"bizName":"","ve|
|00000090| 72 73 69 6f 6e 22 3a 30 2c 22 74 65 6d 70 6c 61 |rsion":0,"templa|
|000000a0| 74 65 49 64 22 3a 30 2c 22 72 6f 75 74 65 53 63 |teId":0,"routeSc|
|000000b0| 68 65 64 75 6c 65 54 79 70 65 22 3a 30 2c 22 74 |heduleType":0,"t|
|000000c0| 72 61 66 66 69 63 4d 61 72 6b 73 22 3a 7b 22 36 |rafficMarks":{"6|
|000000d0| 22 3a 7b 22 65 6e 74 69 74 69 65 73 22 3a 5b 7b |":{"entities":[{|
|000000e0| 22 6e 61 6d 65 22 3a 22 70 74 70 46 6c 61 67 22 |"name":"ptpFlag"|
|000000f0| 2c 22 65 78 74 72 61 49 6e 66 6f 73 22 3a 7b 22 |,"extraInfos":{"|
|00000100| 70 74 70 54 61 73 6b 22 3a 22 32 39 35 39 38 38 |ptpTask":"295988|
|00000110| 22 7d 7d 5d 7d 7d 7d 7d 0d 0a 6b 74 72 61 63 65 |"}}]}}}}..ktrace|
|00000120| 44 65 62 75 67 49 64 3a 20 39 39 39 39 32 35 39 |DebugId: 9999259|
|00000130| 36 32 36 33 35 36 0d 0a 58 2d 4b 54 72 61 63 65 |626356..X-KTrace|
|00000140| 2d 49 64 2d 45 6e 61 62 6c 65 64 3a 20 31 0d 0a |-Id-Enabled: 1..|
|00000150| 61 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 75 73 65 |accept: */*..use|
|00000160| 72 2d 61 67 65 6e 74 3a 20 50 54 50 0d 0a 46 6f |r-agent: PTP..Fo|
|00000170| 72 77 61 72 64 65 64 3a 20 70 72 6f 74 6f 3d 68 |rwarded: proto=h|
|00000180| 74 74 70 3b 68 6f 73 74 3d 22 31 30 2e 39 36 2e |ttp;host="10.96.|
|00000190| 37 30 2e 31 36 32 3a 32 31 32 34 34 22 3b 66 6f |70.162:21244";fo|
|000001a0| 72 3d 22 31 30 2e 34 38 2e 36 33 2e 31 37 37 3a |r="10.48.xxx.xxx:|
|000001b0| 33 36 38 35 36 22 0d 0a 58 2d 46 6f 72 77 61 72 |36856"..X-Forwar|
|000001c0| 64 65 64 2d 46 6f 72 3a 20 31 30 2e 34 38 2e 36 |ded-For: 10.48.6|
|000001d0| 33 2e 31 37 37 0d 0a 58 2d 46 6f 72 77 61 72 64 |3.177..X-Forward|
|000001e0| 65 64 2d 50 72 6f 74 6f 3a 20 68 74 74 70 0d 0a |ed-Proto: http..|
|000001f0| 58 2d 46 6f 72 77 61 72 64 65 64 2d 50 6f 72 74 |X-Forwarded-Port|
|00000200| 3a 20 32 31 32 34 34 0d 0a 58 2d 46 6f 72 77 61 |: 21244..X-Forwa|
|00000210| 72 64 65 64 2d 48 6f 73 74 3a 20 31 30 2e 39 36 |rded-Host: 10.96|
|00000220| 2e 37 30 2e 31 36 32 3a 32 31 32 34 34 0d 0a 68 |.70.162:21244..h|
|00000230| 6f 73 74 3a 20 31 30 2e 31 30 36 2e 31 39 31 2e |ost: 10.106.191.|
|00000240| 32 31 33 3a 32 32 37 31 38 0d 0a 63 6f 6e 74 65 |213:22718..conte|
|00000250| 6e 74 2d 6c 65 6e 67 74 68 3a 20 30 0d 0a 0d 0a |nt-length: 0....|
+--------+-------------------------------------------------+----------------+
2022-10-25 16:25:33.082 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] FLUSH
2022-10-25 16:25:33.082 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 1024, all: 7
2022-10-25 16:25:33.082 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0xac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718]}}, [request_sent])
2022-10-25 16:25:39.039 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [d4f290af-1, L:/10.96.xxx.xxx:21244 - R:/10.48.xxx.xxx:36856] READ COMPLETE
2022-10-25 16:25:39.423 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] READ: 2048B
        +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 54 54 50 2f 31 2e 31 20 32 30 30 20 0d 0a 43 |HTTP/1.1 200 ..C|
|00000010| 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 70 |ontent-Type: app|
|00000020| 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 |lication/json;ch|
|00000030| 61 72 73 65 74 3d 55 54 46 2d 38 0d 0a 43 6f 6e |arset=UTF-8..Con|
|00000040| 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 35 34 |tent-Length: 154|
|00000050| 33 38 38 0d 0a 44 61 74 65 3a 20 54 75 65 2c 20 |388..Date: Tue, |
|00000060| 32 35 20 4f 63 74 20 32 30 32 32 20 30 38 3a 32 |25 Oct 2022 08:2|
|00000070| 35 3a 33 32 20 47 4d 54 0d 0a 0d 0a 7b 22 62 6c |5:32 GMT....{"bl|
|00000080| 6f 63 6b 52 65 73 70 6f 6e 73 65 22 3a 7b 22 72 |ockResponse":{"r|
|00000090| 65 73 75 6c 74 22 3a 30 2c 22 62 6c 6f 63 6b 49 |esult":0,"blockI|
|000000a0| 64 33 30 35 44 61 74 61 22 3a 5b 7b 22 69 6d 61 |d305Data":[{"ima|
.............................................................................
|00000760| 7a 74 6c 63 2f 63 6f 76 65 72 5f 49 42 41 4e 67 |ztlc/cover_IBANg|
|00000770| 52 6e 63 44 79 6f 5f 72 61 77 2e 6a 70 67 22 7d |RncDyo_raw.jpg"}|
|00000780| 2c 22 75 73 65 72 43 6f 76 65 72 49 6d 67 49 6e |,"userCoverImgIn|
|00000790| 66 6f 22 3a 7b 22 77 69 64 74 68 22 3a 31 31 30 |fo":{"width":110|
|000007a0| 2c 22 68 65 69 67 68 74 22 3a 31 31 30 2c 22 73 |,"height":110,"s|
|000007b0| 69 7a 65 22 3a 36 37 32 36 2c 22 74 79 70 65 22 |ize":6726,"type"|
|000007c0| 3a 30 2c 22 61 6e 69 6d 61 74 65 64 22 3a 66 61 |:0,"animated":fa|
|000007d0| 6c 73 65 2c 22 74 68 75 6d 62 6e 61 69 6c 49 6d |lse,"thumbnailIm|
|000007e0| 61 67 65 22 3a 7b 22 63 64 6e 55 72 6c 73 22 3a |age":{"cdnUrls":|
|000007f0| 5b 7b 22 75 72 6c 22 3a 22 68 74 74 70 73 3a 2f |[{"url":"https:/|
+--------+-------------------------------------------------+----------------+
2022-10-25 16:25:39.423 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Received response (auto-read:false) : [Content-Type=application/json;charset=UTF-8, Date=Tue, 25 Oct 2022 08:25:32 GMT, content-length=154388]
2022-10-25 16:25:39.423 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0xac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718]}}, [response_received])
2022-10-25 16:25:39.423 INFO    [reactor-http-epoll-1] (KsNettyWriteResponseFilter.java:87) - NettyWriteResponseFilter start inbound: ac4f0d00, outbound: [d4f290af-2863]
2022-10-25 16:25:39.423 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-10-25 16:25:39.424 INFO    [reactor-http-epoll-1] (Loggers.java:271) - HttpOperations send is not active [id: 0xd4f290af, L:/10.96.xxx.xxx:21244 ! R:/10.48.xxx.xxx:36856]
2022-10-25 16:25:39.424 INFO    [reactor-http-epoll-1] (AbstractServerHttpResponse.java:247) - doOnError: ==============reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
2022-10-25 16:25:39.424 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 2048, all: 98311
2022-10-25 16:25:39.424 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 32768, all: 131079
2022-10-25 16:25:39.541 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] READ: 32768B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 2f 74 78 2d 66 72 65 65 2d 69 6d 67 73 32 2e 61 |/tx-free-imgs2.a|
|00000010| 63 66 75 6e 2e 63 6e 2f 6b 69 6d 67 2f 62 73 32 |cfun.cn/kimg/bs2|
|00000020| 2f 7a 74 2d 69 6d 61 67 65 2d 68 6f 73 74 2f 43 |/zt-image-host/C|
|00000030| 68 51 77 4f 47 5a 6b 59 54 4a 6c 4e 44 41 79 4d |hQwOGZkYTJlNDAyM|
|00000040| 54 42 68 59 6d 4a 6b 4f 54 51 77 4d 68 43 59 7a |TBhYmJkOTQwMhCYz|
|00000050| 4e 63 76 2e 70 6e 67 22 2c 22 66 72 65 65 54 72 |Ncv.png","freeTr|
|00000060| 61 66 66 69 63 50 72 6f 64 75 63 74 41 62 62 72 |afficProductAbbr|
|00000070| 65 76 69 61 74 69 6f 6e 22 3a 22 22 2c 22 66 72 |eviation":"","fr|
.............................................................................
|00007f70| 6e 61 6e 61 4e 75 6d 22 3a 30 2c 22 75 73 65 72 |nanaNum":0,"user|
|00007f80| 43 6f 76 65 72 22 3a 22 68 74 74 70 73 3a 2f 2f |Cover":"https://|
|00007f90| 69 6d 67 73 2e 61 69 78 69 66 61 6e 2e 63 6f 6d |imgs.aixifan.com|
|00007fa0| 2f 73 74 79 6c 65 2f 69 6d 61 67 65 2f 32 30 31 |/style/image/201|
|00007fb0| 39 30 37 2f 70 4f 6c 4d 4c 45 66 64 6a 53 31 70 |907/pOlMLEfdjS1p|
|00007fc0| 49 50 69 58 37 74 43 59 6b 56 64 41 69 79 47 56 |IPiX7tCYkVdAiyGV|
|00007fd0| 30 48 79 48 2e 6a 70 67 22 2c 22 76 65 72 69 66 |0HyH.jpg","verif|
|00007fe0| 69 65 64 22 3a 30 2c 22 76 69 64 65 6f 49 64 22 |ied":0,"videoId"|
|00007ff0| 3a 31 30 37 31 33 35 39 38 2c 22 73 75 62 54 69 |:10713598,"subTi|
+--------+-------------------------------------------------+----------------+
2022-10-25 16:25:39.541 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-25 16:25:39.541 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 65536, all: 196615
2022-10-25 16:25:39.780 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] READ: 65536B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 74 6c 65 22 3a 22 22 2c 22 6c 69 6e 6b 22 3a 22 |tle":"","link":"|
|00000010| 2f 76 2f 61 63 31 30 39 31 34 36 35 39 22 2c 22 |/v/ac10914659","|
|00000020| 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 e5 8a |description":"..|
|00000030| a9 e6 89 8b ef bc 9a e8 80 81 e6 9d bf ef bc 8c |................|
|00000040| e3 80 8a e9 a3 8e e4 ba 91 e3 80 8b e5 b7 b2 e7 |................|
|00000050| bb 8f e6 88 90 e5 8a 9f e5 8d 87 e7 ba a7 e4 b8 |................|
|00000060| ba e6 a0 bc e6 96 97 e6 b8 b8 e6 88 8f ef bc 8c |................|
|00000070| e7 ac ac e4 b8 80 e8 bd ae e5 b0 8f 42 4f 53 53 |............BOSS|
|00000080| e6 98 af e5 89 91 e5 b2 b3 ef bc 8c e6 8e a5 e4 |................|
|00000090| b8 8b e6 9d a5 e6 80 8e e4 b9 88 e5 8a 9e ef bc |................|
|000000a0| 9f 3c 62 72 2f 3e e8 80 81 e9 a9 ac ef bc 9a e6 |.<br/>..........|
|000000b0| 89 93 e5 ae 8c e5 86 8d e8 af b4 e5 90 a7 ef bc |................|
.............................................................................
|0000ff50| 61 4e 75 6d 22 3a 30 2c 22 75 73 65 72 43 6f 76 |aNum":0,"userCov|
|0000ff60| 65 72 22 3a 22 22 2c 22 76 65 72 69 66 69 65 64 |er":"","verified|
|0000ff70| 22 3a 30 2c 22 76 69 64 65 6f 49 64 22 3a 30 2c |":0,"videoId":0,|
|0000ff80| 22 73 75 62 54 69 74 6c 65 22 3a 22 22 2c 22 6c |"subTitle":"","l|
|0000ff90| 69 6e 6b 22 3a 22 68 74 74 70 3a 2f 2f 77 77 77 |ink":"http://www|
|0000ffa0| 2e 61 63 66 75 6e 2e 63 6e 2f 73 65 61 72 63 68 |.acfun.cn/search|
|0000ffb0| 2f 3f 23 70 61 67 65 3d 31 3b 71 75 65 72 79 3d |/?#page=1;query=|
|0000ffc0| 25 45 39 25 38 37 25 39 31 25 45 35 25 38 35 25 |%E9%87%91%E5%85%|
|0000ffd0| 38 39 25 45 35 25 42 45 25 41 31 25 45 34 25 42 |89%E5%BE%A1%E4%B|
|0000ffe0| 39 25 39 44 25 45 37 25 39 35 25 38 43 3b 74 79 |9%9D%E7%95%8C;ty|
|0000fff0| 70 65 3d 76 69 64 65 6f 22 2c 22 64 65 73 63 72 |pe=video","descr|
+--------+-------------------------------------------------+----------------+
2022-10-25 16:25:39.780 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-25 16:26:08.223 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [d4f290af-1, L:/10.96.xxx.xxx:21244 ! R:/10.48.xxx.xxx:36856] INACTIVE
2022-10-25 16:26:08.223 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [d4f290af, L:/10.96.xxx.xxx:21244 ! R:/10.48.xxx.xxx:36856] UNREGISTERED
2022-10-25 16:26:08.624 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] READ: 54160B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 69 70 74 69 6f 6e 22 3a 22 22 2c 22 72 65 61 6c |iption":"","real|
|00000010| 6d 4e 61 6d 65 22 3a 22 22 2c 22 65 6e 64 54 69 |mName":"","endTi|
|00000020| 6d 65 22 3a 30 2c 22 75 73 65 72 4e 61 6d 65 22 |me":0,"userName"|
|00000030| 3a 22 22 2c 22 6d 65 64 69 61 54 79 70 65 22 3a |:"","mediaType":|
|00000040| 38 7d 2c 7b 22 69 6d 61 67 65 49 6e 66 6f 22 3a |8},{"imageInfo":|
|00000050| 6e 75 6c 6c 2c 22 75 73 65 72 43 6f 76 65 72 49 |null,"userCoverI|
|00000060| 6d 67 49 6e 66 6f 22 3a 6e 75 6c 6c 2c 22 73 6f |mgInfo":null,"so|
|00000070| 72 74 22 3a 30 2c 22 74 69 6d 65 22 3a 30 2c 22 |rt":0,"time":0,"|
|00000080| 69 64 22 3a 31 35 31 30 32 31 2c 22 63 6f 6e 74 |id":151021,"cont|
|00000090| 65 6e 74 22 3a 22 22 2c 22 63 72 65 61 74 6f 72 |ent":"","creator|
.............................................................................
|0000d310| 2c 22 6d 6f 64 75 6c 65 49 6e 64 65 78 22 3a 30 |,"moduleIndex":0|
|0000d320| 2c 22 63 6f 6e 74 65 6e 74 43 6f 75 6e 74 22 3a |,"contentCount":|
|0000d330| 33 2c 22 75 70 64 61 74 65 72 49 64 22 3a 31 2c |3,"updaterId":1,|
|0000d340| 22 69 73 44 65 6c 65 74 65 64 22 3a 30 2c 22 6f |"isDeleted":0,"o|
|0000d350| 6e 65 70 61 67 65 43 6f 75 6e 74 22 3a 34 2c 22 |nepageCount":4,"|
|0000d360| 6f 72 64 65 72 73 22 3a 35 2c 22 73 75 62 54 69 |orders":5,"subTi|
|0000d370| 74 6c 65 22 3a 22 22 2c 22 6c 69 6e 6b 22 3a 22 |tle":"","link":"|
|0000d380| 22 2c 22 73 74 61 74 75 73 22 3a 30 7d 5d 7d 7d |","status":0}]}}|
+--------+-------------------------------------------------+----------------+
2022-10-25 16:26:08.624 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Received last HTTP packet
2022-10-25 16:26:08.624 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0xac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718]}}, [response_completed])
2022-10-25 16:26:08.624 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0xac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718]}}, [disconnecting])
2022-10-25 16:26:08.624 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Releasing channel
2022-10-25 16:26:08.624 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Channel cleaned, now: 102 active connections, 2 inactive connections and 0 pending acquire requests.
2022-10-25 16:26:08.624 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 65536, all: 6750215
2022-10-25 16:26:08.624 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 65536, all: 6684679
2022-10-25 16:26:08.624 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-25 16:26:09.085 ERROR   [reactor-http-epoll-1] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        Hint: [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Buffered ByteBufHolder in the inbound buffer queue
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
        reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
        reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1561)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1567)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#4:
        Hint: 'reactor.left.loggingHandler' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#5:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
Created at:
        io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:480)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
: 15 leak records were discarded because they were duplicates
: 131080 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.
2022-10-25 16:26:09.089 ERROR   [reactor-http-epoll-1] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        Hint: [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Buffered ByteBufHolder in the inbound buffer queue
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:25)
        reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
        reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1567)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1561)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#4:
        Hint: 'reactor.left.loggingHandler' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#5:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
Created at:
        io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
: 13 leak records were discarded because they were duplicates
: 65534 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.
2022-10-25 16:26:09.090 ERROR   [reactor-http-epoll-1] (ResourceLeakDetector.java:319) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        Hint: [ac4f0d00-1, L:/10.96.xxx.xxx:50070 - R:10.106.191.213/10.106.191.213:22718] Buffered ByteBufHolder in the inbound buffer queue
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
        io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
        io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
        reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:384)
        reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:685)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1567)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:161)
        io.netty.buffer.ByteBufUtil$HexUtil.appendPrettyHexDump(ByteBufUtil.java:1561)
        io.netty.buffer.ByteBufUtil$HexUtil.access$500(ByteBufUtil.java:1420)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1416)
        io.netty.buffer.ByteBufUtil.appendPrettyHexDump(ByteBufUtil.java:1407)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.formatByteBuf(ReactorNettyLoggingHandler.java:231)
        reactor.netty.transport.logging.ReactorNettyLoggingHandler.format(ReactorNettyLoggingHandler.java:134)
        io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#4:
        Hint: 'reactor.left.loggingHandler' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
#5:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
Created at:
        io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
        io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:829)
: 15 leak records were discarded because they were duplicates
: 108325 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.
2022-10-25 16:26:14.798 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00, L:/10.96.xxx.xxx:50070 ! R:10.106.191.213/10.106.191.213:22718] INACTIVE
2022-10-25 16:26:14.798 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [ac4f0d00, L:/10.96.xxx.xxx:50070 ! R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0xac4f0d00, L:/10.96.xxx.xxx:50070 ! R:10.106.191.213/10.106.191.213:22718]}, [disconnecting])
2022-10-25 16:26:14.798 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [ac4f0d00, L:/10.96.xxx.xxx:50070 ! R:10.106.191.213/10.106.191.213:22718] UNREGISTERED
yanwenliang commented 1 year ago

when I move https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L69 to https://github.com/spring-cloud/spring-cloud-gateway/blob/62e0f241d2aac61e699e4733ff2ed46c6865b045/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/NettyWriteResponseFilter.java#L102 and add log.

return chain.filter(exchange)
        .then(Mono.defer(() -> {
            Connection connection = exchange.getAttribute(CLIENT_RESPONSE_CONN_ATTR);
                 ......
            return (isStreamingMediaType(contentType)
                    ? response.writeAndFlushWith(body.map(Flux::just))
                    : response.writeWith(body));
        })).doOnCancel(() -> cleanup(exchange))
        .doOnError(throwable -> {
            log.error("NettyWriteResponseFilter =================== filter doOnError", throwable);
            cleanup(exchange);
        });

and change the cleanup code, and add some log:

private void cleanup(ServerWebExchange exchange) {
        Connection connection = exchange.getAttribute(CLIENT_RESPONSE_CONN_ATTR);
                log.info("NettyWriteResponseFilter =================== cleanup connection {}", connection);
                if (connection != null && connection.channel().isActive()) {
                        log.info("NettyWriteResponseFilter =================== cleanup connection channel {}", connection.channel());
                    connection.dispose();
                }
        }

the memory leak disappear. The full log is:

2022-10-29 03:07:59.586 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [1b174349, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] New http connection, requesting read
2022-10-29 03:07:59.586 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [1b174349, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-10-29 03:07:59.586 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [1b174349, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] REGISTERED
2022-10-29 03:07:59.586 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [1b174349, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] ACTIVE
2022-10-29 03:08:00.618 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [1b174349, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] READ: 394B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 72 65 73 74 2f 77 64 2f 62 65 6e |GET /rest/wd/ben|
|00000010| 63 68 6d 61 72 6b 2f 72 65 73 70 6f 6e 73 65 20 |chmark/response |
|00000020| 48 54 54 50 2f 31 2e 31 0d 0a 70 74 70 2d 66 6c |HTTP/1.1..ptp-fl|
|00000030| 61 67 3a 20 31 0d 0a 6d 69 6e 54 69 6d 65 3a 20 |ag: 1..minTime: |
|00000040| 30 0d 0a 6d 61 78 54 69 6d 65 3a 20 30 0d 0a 74 |0..maxTime: 0..t|
|00000050| 72 61 63 65 2d 63 6f 6e 74 65 78 74 3a 20 7b 22 |race-context: {"|
|00000060| 6c 61 6e 65 49 64 22 3a 22 22 2c 22 73 74 72 65 |laneId":"","stre|
|00000070| 73 73 54 65 73 74 43 6f 6e 74 65 78 74 22 3a 7b |ssTestContext":{|
|00000080| 22 62 69 7a 4e 61 6d 65 22 3a 22 22 2c 22 76 65 |"bizName":"","ve|
|00000090| 72 73 69 6f 6e 22 3a 30 2c 22 74 65 6d 70 6c 61 |rsion":0,"templa|
|000000a0| 74 65 49 64 22 3a 30 2c 22 72 6f 75 74 65 53 63 |teId":0,"routeSc|
|000000b0| 68 65 64 75 6c 65 54 79 70 65 22 3a 30 2c 22 74 |heduleType":0,"t|
|000000c0| 72 61 66 66 69 63 4d 61 72 6b 73 22 3a 7b 22 36 |rafficMarks":{"6|
|000000d0| 22 3a 7b 22 65 6e 74 69 74 69 65 73 22 3a 5b 7b |":{"entities":[{|
|000000e0| 22 6e 61 6d 65 22 3a 22 70 74 70 46 6c 61 67 22 |"name":"ptpFlag"|
|000000f0| 2c 22 65 78 74 72 61 49 6e 66 6f 73 22 3a 7b 22 |,"extraInfos":{"|
|00000100| 70 74 70 54 61 73 6b 22 3a 22 32 39 37 39 33 33 |ptpTask":"297933|
|00000110| 22 7d 7d 5d 7d 7d 7d 7d 0d 0a 6b 74 72 61 63 65 |"}}]}}}}..ktrace|
|00000120| 44 65 62 75 67 49 64 3a 20 39 39 39 39 32 34 32 |DebugId: 9999242|
|00000130| 39 34 32 35 30 34 0d 0a 58 2d 4b 54 72 61 63 65 |942504..X-KTrace|
|00000140| 2d 49 64 2d 45 6e 61 62 6c 65 64 3a 20 31 0d 0a |-Id-Enabled: 1..|
|00000150| 68 6f 73 74 3a 20 31 30 2e 39 36 2e 37 30 2e 31 |host: 10.96.70.1|
|00000160| 36 32 3a 32 31 34 31 39 0d 0a 61 63 63 65 70 74 |62:21419..accept|
|00000170| 3a 20 2a 2f 2a 0d 0a 75 73 65 72 2d 61 67 65 6e |: */*..user-agen|
|00000180| 74 3a 20 50 54 50 0d 0a 0d 0a                   |t: PTP....      |
+--------+-------------------------------------------------+----------------+
2022-10-29 03:08:00.618 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 2048, all: 7
2022-10-29 03:08:00.619 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [1b174349, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] Increasing pending responses, now 1
2022-10-29 03:08:00.619 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [1b174349-1, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@7d2e086c
2022-10-29 03:08:00.620 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526] Created a new pooled channel, now: 0 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-29 03:08:00.620 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526] Initialized pipeline DefaultChannelPipeline{(reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-10-29 03:08:00.620 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526] REGISTERED
2022-10-29 03:08:00.620 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [1b174349-1, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] READ COMPLETE
2022-10-29 03:08:03.007 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [958c9526] Connecting to [/10.106.191.213:22718].
2022-10-29 03:08:03.007 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526] CONNECT: /10.106.191.213:22718
2022-10-29 03:08:04.772 DEBUG   [reactor-http-epoll-1] (Loggers.java:246) - [958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] Registering pool release on close event for channel
2022-10-29 03:08:04.772 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] Channel connected, now: 154 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-29 03:08:04.772 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] ACTIVE
2022-10-29 03:08:04.772 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] onStateChange(PooledConnection{channel=[id: 0x958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718]}, [connected])
2022-10-29 03:08:04.773 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/, connection=PooledConnection{channel=[id: 0x958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718]}}, [configured])
2022-10-29 03:08:04.773 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] Handler is being applied: {uri=http://10.106.191.213:22718/rest/wd/benchmark/response, method=GET}
2022-10-29 03:08:04.773 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718]}}, [request_prepared])
2022-10-29 03:08:04.773 TRACE   [reactor-http-epoll-1] (NettyRoutingFilter.java:143) - outbound route: 958c9526, inbound: [1b174349-255]
2022-10-29 03:08:04.773 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [1b174349-1, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] FluxReceive{pending=0, cancelled=false, inboundDone=true, inboundError=null}: subscribing inbound receiver
2022-10-29 03:08:04.773 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 256, all: 263
2022-10-29 03:08:04.773 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 256, all: 519
2022-10-29 03:08:04.774 INFO    [reactor-http-epoll-1] (PlatformDependent.java:811) - incrementMemoryCounter: 512, all: 1031
2022-10-29 03:08:04.782 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] WRITE: 606B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 72 65 73 74 2f 77 64 2f 62 65 6e |GET /rest/wd/ben|
|00000010| 63 68 6d 61 72 6b 2f 72 65 73 70 6f 6e 73 65 20 |chmark/response |
|00000020| 48 54 54 50 2f 31 2e 31 0d 0a 70 74 70 2d 66 6c |HTTP/1.1..ptp-fl|
|00000030| 61 67 3a 20 31 0d 0a 6d 69 6e 54 69 6d 65 3a 20 |ag: 1..minTime: |
|00000040| 30 0d 0a 6d 61 78 54 69 6d 65 3a 20 30 0d 0a 74 |0..maxTime: 0..t|
|00000050| 72 61 63 65 2d 63 6f 6e 74 65 78 74 3a 20 7b 22 |race-context: {"|
|00000060| 6c 61 6e 65 49 64 22 3a 22 22 2c 22 73 74 72 65 |laneId":"","stre|
|00000070| 73 73 54 65 73 74 43 6f 6e 74 65 78 74 22 3a 7b |ssTestContext":{|
|00000080| 22 62 69 7a 4e 61 6d 65 22 3a 22 22 2c 22 76 65 |"bizName":"","ve|
|00000090| 72 73 69 6f 6e 22 3a 30 2c 22 74 65 6d 70 6c 61 |rsion":0,"templa|
|000000a0| 74 65 49 64 22 3a 30 2c 22 72 6f 75 74 65 53 63 |teId":0,"routeSc|
|000000b0| 68 65 64 75 6c 65 54 79 70 65 22 3a 30 2c 22 74 |heduleType":0,"t|
|000000c0| 72 61 66 66 69 63 4d 61 72 6b 73 22 3a 7b 22 36 |rafficMarks":{"6|
|000000d0| 22 3a 7b 22 65 6e 74 69 74 69 65 73 22 3a 5b 7b |":{"entities":[{|
|000000e0| 22 6e 61 6d 65 22 3a 22 70 74 70 46 6c 61 67 22 |"name":"ptpFlag"|
|000000f0| 2c 22 65 78 74 72 61 49 6e 66 6f 73 22 3a 7b 22 |,"extraInfos":{"|
|00000100| 70 74 70 54 61 73 6b 22 3a 22 32 39 37 39 33 33 |ptpTask":"297933|
|00000110| 22 7d 7d 5d 7d 7d 7d 7d 0d 0a 6b 74 72 61 63 65 |"}}]}}}}..ktrace|
|00000120| 44 65 62 75 67 49 64 3a 20 39 39 39 39 32 34 32 |DebugId: 9999242|
|00000130| 39 34 32 35 30 34 0d 0a 58 2d 4b 54 72 61 63 65 |942504..X-KTrace|
|00000140| 2d 49 64 2d 45 6e 61 62 6c 65 64 3a 20 31 0d 0a |-Id-Enabled: 1..|
|00000150| 61 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 75 73 65 |accept: */*..use|
|00000160| 72 2d 61 67 65 6e 74 3a 20 50 54 50 0d 0a 46 6f |r-agent: PTP..Fo|
|00000170| 72 77 61 72 64 65 64 3a 20 70 72 6f 74 6f 3d 68 |rwarded: proto=h|
|00000180| 74 74 70 3b 68 6f 73 74 3d 22 31 30 2e 39 36 2e |ttp;host="10.96.|
|00000190| 37 30 2e 31 36 32 3a 32 31 34 31 39 22 3b 66 6f |70.162:21419";fo|
|000001a0| 72 3d 22 31 30 2e 33 34 2e 36 30 2e 33 33 3a 34 |r="10.34.xxx.xxx:4|
|000001b0| 39 35 37 30 22 0d 0a 58 2d 46 6f 72 77 61 72 64 |9570"..X-Forward|
|000001c0| 65 64 2d 46 6f 72 3a 20 31 30 2e 33 34 2e 36 30 |ed-For: 10.34.60|
|000001d0| 2e 33 33 0d 0a 58 2d 46 6f 72 77 61 72 64 65 64 |.33..X-Forwarded|
|000001e0| 2d 50 72 6f 74 6f 3a 20 68 74 74 70 0d 0a 58 2d |-Proto: http..X-|
|000001f0| 46 6f 72 77 61 72 64 65 64 2d 50 6f 72 74 3a 20 |Forwarded-Port: |
|00000200| 32 31 34 31 39 0d 0a 58 2d 46 6f 72 77 61 72 64 |21419..X-Forward|
|00000210| 65 64 2d 48 6f 73 74 3a 20 31 30 2e 39 36 2e 37 |ed-Host: 10.96.7|
|00000220| 30 2e 31 36 32 3a 32 31 34 31 39 0d 0a 68 6f 73 |0.162:21419..hos|
|00000230| 74 3a 20 31 30 2e 31 30 36 2e 31 39 31 2e 32 31 |t: 10.106.191.21|
|00000240| 33 3a 32 32 37 31 38 0d 0a 63 6f 6e 74 65 6e 74 |3:22718..content|
|00000250| 2d 6c 65 6e 67 74 68 3a 20 30 0d 0a 0d 0a       |-length: 0....  |
+--------+-------------------------------------------------+----------------+
2022-10-29 03:08:04.782 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] FLUSH
2022-10-29 03:08:04.782 INFO    [reactor-http-epoll-1] (PlatformDependent.java:819) - decrementMemoryCounter: 1024, all: 7
2022-10-29 03:08:04.782 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718]}}, [request_sent])
2022-10-29 03:08:11.970 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [1b174349-1, L:/10.96.xxx.xxx:21419 - R:/10.34.xxx.xxx:49570] READ COMPLETE
2022-10-29 03:08:11.989 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] READ: 2048B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 54 54 50 2f 31 2e 31 20 32 30 30 20 0d 0a 43 |HTTP/1.1 200 ..C|
|00000010| 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 70 |ontent-Type: app|
|00000020| 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 |lication/json;ch|
|00000030| 61 72 73 65 74 3d 55 54 46 2d 38 0d 0a 43 6f 6e |arset=UTF-8..Con|
|00000040| 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 35 34 |tent-Length: 154|
|00000050| 33 38 38 0d 0a 44 61 74 65 3a 20 46 72 69 2c 20 |388..Date: Fri, |
|00000060| 32 38 20 4f 63 74 20 32 30 32 32 20 31 39 3a 30 |28 Oct 2022 19:0|
|00000070| 38 3a 30 34 20 47 4d 54 0d 0a 0d 0a 7b 22 62 6c |8:04 GMT....{"bl|
|00000080| 6f 63 6b 52 65 73 70 6f 6e 73 65 22 3a 7b 22 72 |ockResponse":{"r|
|00000090| 65 73 75 6c 74 22 3a 30 2c 22 62 6c 6f 63 6b 49 |esult":0,"blockI|
.............................................................................
|00000790| 66 6f 22 3a 7b 22 77 69 64 74 68 22 3a 31 31 30 |fo":{"width":110|
|000007a0| 2c 22 68 65 69 67 68 74 22 3a 31 31 30 2c 22 73 |,"height":110,"s|
|000007b0| 69 7a 65 22 3a 36 37 32 36 2c 22 74 79 70 65 22 |ize":6726,"type"|
|000007c0| 3a 30 2c 22 61 6e 69 6d 61 74 65 64 22 3a 66 61 |:0,"animated":fa|
|000007d0| 6c 73 65 2c 22 74 68 75 6d 62 6e 61 69 6c 49 6d |lse,"thumbnailIm|
|000007e0| 61 67 65 22 3a 7b 22 63 64 6e 55 72 6c 73 22 3a |age":{"cdnUrls":|
|000007f0| 5b 7b 22 75 72 6c 22 3a 22 68 74 74 70 73 3a 2f |[{"url":"https:/|
+--------+-------------------------------------------------+----------------+
2022-10-29 03:08:11.990 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] Received response (auto-read:false) : [Content-Type=application/json;charset=UTF-8, Date=Fri, 28 Oct 2022 19:08:04 GMT, content-length=154388]
2022-10-29 03:08:11.990 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718]}}, [response_received])
2022-10-29 03:08:11.990 INFO    [reactor-http-epoll-1] (NettyWriteResponseFilter.java:83) - NettyWriteResponseFilter start inbound: 958c9526, outbound: [1b174349-255]
2022-10-29 03:08:11.990 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-10-29 03:08:11.991 INFO    [reactor-http-epoll-1] (Loggers.java:271) - HttpOperations send is not active [id: 0x1b174349, L:/10.96.xxx.xxx:21419 ! R:/10.34.xxx.xxx:49570]
2022-10-29 03:08:11.991 INFO    [reactor-http-epoll-1] (AbstractServerHttpResponse.java:247) - doOnError: ==============reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
2022-10-29 03:08:11.992 ERROR   [reactor-http-epoll-1] (NettyWriteResponseFilter.java:102) - NettyWriteResponseFilter =================== filter doOnError

reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
        at reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:59) ~[reactor-netty-core-1.0.16.jar!/:1.0.16]
        at reactor.netty.http.HttpOperations.then(HttpOperations.java:172) ~[classes!/:1.0.16]
        at reactor.netty.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:702) ~[reactor-netty-core-1.0.16.jar!/:1.0.16]
        at reactor.netty.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:695) ~[reactor-netty-core-1.0.16.jar!/:1.0.16]
        at reactor.netty.NettyOutbound.then(NettyOutbound.java:358) ~[reactor-netty-core-1.0.16.jar!/:1.0.16]
        at reactor.netty.http.HttpOperations.send(HttpOperations.java:110) ~[classes!/:1.0.16]
        at org.springframework.http.server.reactive.ReactorServerHttpResponse.writeWithInternal(ReactorServerHttpResponse.java:92) ~[spring-web-5.3.17.jar!/:5.3.17]
        at org.springframework.http.server.reactive.AbstractServerHttpResponse.lambda$writeWith$7(AbstractServerHttpResponse.java:245) ~[classes!/:5.3.17]
        at org.springframework.http.server.reactive.AbstractServerHttpResponse.doCommit(AbstractServerHttpResponse.java:305) ~[classes!/:5.3.17]
        at org.springframework.http.server.reactive.AbstractServerHttpResponse.lambda$writeWith$8(AbstractServerHttpResponse.java:245) ~[classes!/:5.3.17]
        at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onNext(ChannelSendOperator.java:186) ~[spring-web-5.3.17.jar!/:5.3.17]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.15.jar!/:3.4.15]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.4.15.jar!/:3.4.15]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.15.jar!/:3.4.15]
        at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:279) ~[reactor-netty-core-1.0.16.jar!/:1.0.16]
        at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:388) ~[reactor-netty-core-1.0.16.jar!/:1.0.16]
        at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404) ~[reactor-netty-core-1.0.16.jar!/:1.0.16]
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:724) ~[reactor-netty-http-1.0.16.jar!/:1.0.16]
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93) ~[reactor-netty-core-1.0.16.jar!/:1.0.16]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[netty-codec-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[netty-codec-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[netty-handler-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-classes-epoll-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar!/:4.1.75.Final]
        at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

2022-10-29 03:08:11.993 INFO    [reactor-http-epoll-1] (NettyWriteResponseFilter.java:126) - NettyWriteResponseFilter =================== cleanup connection GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718]}}
2022-10-29 03:08:11.993 INFO    [reactor-http-epoll-1] (NettyWriteResponseFilter.java:128) - NettyWriteResponseFilter =================== cleanup connection channel [id: 0x958c9526, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718]
2022-10-29 03:08:11.993 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526-1, L:/10.96.xxx.xxx:42490 - R:10.106.191.213/10.106.191.213:22718] CLOSE
2022-10-29 03:08:11.993 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 ! R:10.106.191.213/10.106.191.213:22718] Channel closed, now: 587 active connections, 0 inactive connections and 0 pending acquire requests.
2022-10-29 03:08:11.993 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526-1, L:/10.96.xxx.xxx:42490 ! R:10.106.191.213/10.106.191.213:22718] CLOSE
2022-10-29 03:08:11.996 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526-1, L:/10.96.xxx.xxx:42490 ! R:10.106.191.213/10.106.191.213:22718] READ COMPLETE
2022-10-29 03:08:18.861 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [1b174349-1, L:/10.96.xxx.xxx:21419 ! R:/10.34.xxx.xxx:49570] INACTIVE
2022-10-29 03:08:18.861 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [1b174349, L:/10.96.xxx.xxx:21419 ! R:/10.34.xxx.xxx:49570] UNREGISTERED
2022-10-29 03:08:18.876 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526-1, L:/10.96.xxx.xxx:42490 ! R:10.106.191.213/10.106.191.213:22718] INACTIVE
2022-10-29 03:08:18.876 DEBUG   [reactor-http-epoll-1] (Loggers.java:251) - [958c9526-1, L:/10.96.xxx.xxx:42490 ! R:10.106.191.213/10.106.191.213:22718] onStateChange(GET{uri=/rest/wd/benchmark/response, connection=PooledConnection{channel=[id: 0x958c9526, L:/10.96.xxx.xxx:42490 ! R:10.106.191.213/10.106.191.213:22718]}}, [disconnecting])
2022-10-29 03:08:18.876 DEBUG   [reactor-http-epoll-1] (AbstractInternalLogger.java:147) - [958c9526-1, L:/10.96.xxx.xxx:42490 ! R:10.106.191.213/10.106.191.213:22718] UNREGISTERED
yanwenliang commented 1 year ago

@violetagg you can see this PR https://github.com/spring-cloud/spring-cloud-gateway/pull/2765 Thanks.

violetagg commented 1 year ago

@yanwenliang I'm closing this one. For the time being I don't see an issue in Reactor Netty.

yanwenliang commented 1 year ago

@yanwenliang I'm closing this one. For the time being I don't see an issue in Reactor Netty.

@violetagg Ok, Thanks! I will create an issue for spring cloud gateway.