search

reactor / reactor-netty

TCP/HTTP/UDP/QUIC client/server with Reactor over Netty
https://projectreactor.io
Apache License 2.0
2.6k stars 645 forks source link

LEAK: ByteBuf.release() was not called before it's garbage-collected. #3127

Closed subham611 closed 6 months ago

subham611 commented 7 months ago

Expected Behavior

I am using spring boot 2.6.13 version with spring-boot-starter-webflux dependency. During low request count spring server works fine but when rps increased I am facing issue related to BUFFER LEAK.

Actual Behavior

Getting below mentioned error: LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at:io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116) org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:71) org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39) org.springframework.core.codec.CharSequenceEncoder.encodeValue(CharSequenceEncoder.java:91) org.springframework.core.codec.CharSequenceEncoder.lambda$encode$0(CharSequenceEncoder.java:75) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113) reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103) reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2731) reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2194) reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2068) reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96) reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:200) reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:251) reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:336) reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398) reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:325) reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234) reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onError(MonoIgnoreElements.java:84) reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onError(FluxConcatArray.java:207) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:142) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:503) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:62) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:258) reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:78) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136) reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:110) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onSubscribe(FluxContextWrite.java:101) reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2664) reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152) reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103) reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2731) reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2194) reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2068) reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96) reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:200) reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249) reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:292) reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:251) reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:336) reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101) reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400) reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419) reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:603) reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

violetagg commented 7 months ago

@subham611 Please follow this FAQ and provide the extended logs https://projectreactor.io/docs/netty/release/reference/index.html#faq.memory-leaks

violetagg commented 7 months ago

@subham611 Consider checking Spring Boot supported versions https://spring.io/projects/spring-boot#support

github-actions[bot] commented 7 months ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

subham611 commented 6 months ago

I updated spring boot to latest version 3.2.4 still getting this error. I have added following jvm parameters, and not seeing any additional log

-XX:MaxDirectMemorySize=1024M
-Dio.netty.leakDetection.targetRecords=40
-Dio.netty.leakDetection.level=paranoid
-Dlogging.level._reactor.netty.channel.LeakDetection=debug

Do I need to set logging.level._reactor.netty.channel.LeakDetection=debug in application.properties.

violetagg commented 6 months ago

@subham611 Try to set it in application.properties

subham611 commented 6 months ago

yeah, saw this https://github.com/reactor/reactor-netty/issues/1874#issuecomment-963266196 Setting following properties in application.properties file

spring.netty.leak-detection=paranoid
logging.level._reactor.netty.channel.LeakDetection=DEBUG
subham611 commented 6 months ago

@violetagg I set above mentioned spring properties but not seeing any change in log message

{"timestamp":1713180562148,"thread":"reactor-http-epoll-1","logger":"io.netty.util.ResourceLeakDetector","message":"LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
    io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
    org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:72)
    org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39)
    org.springframework.core.codec.CharSequenceEncoder.encodeValue(CharSequenceEncoder.java:90)
    org.springframework.core.codec.CharSequenceEncoder.lambda$encode$0(CharSequenceEncoder.java:74)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
    reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
    reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103)
    reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2909)
    reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
    reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241)
    reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
    reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
    reactor.core.publisher.Mono.subscribe(Mono.java:4568)
    reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:202)
    reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
    reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:63)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
    reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:297)
    reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:478)
    reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
    reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:470)
    reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
    reactor.core.publisher.MonoZip$ZipCoordinator.request(MonoZip.java:220)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
    reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
    reactor.core.publisher.MonoZip.subscribe(MonoZip.java:129)
    reactor.core.publisher.Mono.subscribe(Mono.java:4568)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
    reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
    reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
    reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
    reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
    reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234)
    reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
    reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onError(MonoIgnoreElements.java:84)
    reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onError(FluxConcatArray.java:208)
    reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
    reactor.netty.FutureMono$FutureSubscription.operationComplete(FutureMono.java:193)
    io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
    io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
    io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
    io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
    io.netty.util.concurrent.PromiseCombiner.tryPromise(PromiseCombiner.java:170)
    io.netty.util.concurrent.PromiseCombiner.access$600(PromiseCombiner.java:35)
    io.netty.util.concurrent.PromiseCombiner$1.operationComplete0(PromiseCombiner.java:62)
    io.netty.util.concurrent.PromiseCombiner$1.operationComplete(PromiseCombiner.java:44)
    io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
    io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
    io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
    io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
    io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754)
    io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339)
    io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:691)
    io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:735)
    io.netty.channel.AbstractChannel$AbstractUnsafe.handleWriteError(AbstractChannel.java:950)
    io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:933)
    io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:570)
    io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
violetagg commented 6 months ago

@subham611 It doesn't seem that the changes are applied.

violetagg commented 6 months ago

You should be able to see output similar to the one here https://github.com/reactor/reactor-netty/issues/1874#issuecomment-963266196

subham611 commented 6 months ago

Is there any other way to set netty property?

violetagg commented 6 months ago

@subham611 Please consult with the reference documentation of the technologies that you use. For Spring Boot - it should be application.properties (there might be different versions of this - for testing, production etc), then the environment where you run your application - whether the provided JVM OPTS are applied etc.

I cannot help without knowing the environment where you run your app.

You should be able to check the exact JVM OPTS that are used to run your app and the binaries that are published.

subham611 commented 6 months ago

I have added this flag -Dlogging.level.io.netty.util.ResourceLeakDetector=debug as well and i do see following logs

{"timestamp":1713191024585,"thread":"main","logger":"io.netty.util.ResourceLeakDetector","message":"-Dio.netty.leakDetection.level: paranoid","context":"default","log_level":"DEBUG","is_logging_enabled":false}
{"timestamp":1713191024585,"thread":"main","logger":"io.netty.util.ResourceLeakDetector","message":"-Dio.netty.leakDetection.targetRecords: 40","context":"default","log_level":"DEBUG","is_logging_enabled":false}

Which confirms that leakDetection level is set to paranoid but somehow I am not seeing full logs during error.

I have setup a rate limiter using webFilter and I am suspecting some issue there.

@Component
public class RateLimiterFilter implements WebFilter {
  private final RateLimiter rateLimiter;
  @Override
    public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
       if(!rateLimiter.acquirePermission()) {
           log.warn("Too many requests");
           exchange.getResponse().setStatusCode(HttpStatus.TOO_MANY_REQUESTS);
           return Mono.empty(); // I am suspecting this might be causing some issue. 
       }
       return chain.filter(exchange);
    }
}

Let me remove this and try. ref: https://github.com/reactor/reactor-netty/issues/518#issuecomment-440663787

github-actions[bot] commented 6 months ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

subham611 commented 6 months ago

Above change didn't help. I tried setting ResourceLeakDetector property programmatically, that didn't help either

 @Bean
    public void nettyServerCustomizer(){
        ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.PARANOID);
        if(ResourceLeakDetector.isEnabled()) {
            System.out.println(ResourceLeakDetector.getLevel());
        }
    }
violetagg commented 6 months ago

@subham611 Please provide a reproducible example so that we can investigate further.

github-actions[bot] commented 6 months ago

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

github-actions[bot] commented 6 months ago

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.