search

micronaut-projects / micronaut-core

Micronaut Application Framework
http://micronaut.io
Apache License 2.0
6.03k stars 1.05k forks source link

Potential Memory Leak #11007

Open sdelamo opened 1 month ago

sdelamo commented 1 month ago

Issue description

A user upgrading from 3.9.2 to 4 pointed to a potential memory leak

Our application (running on micronaut-platform 4.4.5) is crashing with an out of memory error.

After some analysis of the heap dump, we've found that the issue is due to DirectByteBuffers not being released properly.

We've then enabled this flag to track netty byte buffers:

-Dio.netty.leakDetectionLevel=paranoid

The logs seems to report that the error happens while writing a json response in the micronaut http server:

2024-07-17 13:15:27.048 Jul-17 11:15:27.046 [default-nioEventLoopGroup-5-9] 86.144.114.222 ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.

Recent access records: 

#1:
    io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:617)
    io.netty.buffer.ByteBufOutputStream.write(ByteBufOutputStream.java:80)
    com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2210)
    com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1234)
    com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4804)
    com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3999)
    io.micronaut.jackson.databind.JacksonDatabindMapper.writeValue(JacksonDatabindMapper.java:207)
    io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:161)
    io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:377)
    io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:355)
    io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:249)
    io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:94)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
    io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
    reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
    reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2812)
    reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
    reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
    io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
    io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
    io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
    io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
    io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
    io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
    io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
    io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
    io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
    io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
    io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:840)
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
    io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
    io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:158)
    io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:377)
    io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:355)
    io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:249)
    io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:94)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
    io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
    reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
    reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2812)
    reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
    reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
    io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
    io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
    io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
    io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
    io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
    io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
    io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)

We've tried to update to the latest available micronaut version 4.5.0, to check if the issue was solved already, and run the application in a separate environment, but the leak is still detected in the same place.

2024-07-15 15:58:23.630 Jul-15 13:58:23.629 [io-executor-thread-6] 82.61.68.148 ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.

Recent access records: 

#1:
    io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:617)
    io.netty.buffer.ByteBufOutputStream.write(ByteBufOutputStream.java:80)
    com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2210)
    com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1234)
    com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4804)
    com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3999)
    io.micronaut.jackson.databind.JacksonDatabindMapper.writeValue(JacksonDatabindMapper.java:207)
    io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:163)
    io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:380)
    io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:358)
    io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:248)
    io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:101)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
    io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
    reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
    reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2834)
    reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
    reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
    io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
    io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
    io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
    io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
    io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
    io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
    io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
    io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
    io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
    io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
    io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:840)
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
    io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
    io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:160)
    io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:380)
    io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:358)
    io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:248)
    io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:101)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
    io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
    reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
    reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
    reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2834)
    reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
    reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
    io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
    io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
    io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
    io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
    io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
    io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
    io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
    io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
    io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)

The same request doesn't always produce the log, but we see a steady increase in the size of the memory used by the application.

We are using jackson-databind for serialization and we cannot move to the new micronaut serialization approach just yet.

Inspecting the micronaut codebase, we've found that the buffer allocated in https://github.com/micronaut-projects/micronaut-core/blob/df3a78fd20091e6a773369fc0e942bb51d806b76/http-netty/src/main/java/io/micronaut/http/netty/body/NettyJsonHandler.java#L160 is not released immediately but this is delegated to the DefaultFullHttpResponse#release() method but we where not able to track it further.

graemerocher commented 1 month ago

We will need an example application

sdelamo commented 1 month ago

@yawkat when creating the ByteBufOutputStream, we are not using the constructor which releases on close . I assume that it is intentional because we pass the ByteBuff to DefaultFullHttpResponse and we expect someone to invoke DefaultFullHttpResponse::release. Who is doing that?

yawkat commented 1 month ago

Yes it's intentional. PipeliningServerHandler does the release. This is not sufficient information to determine the cause of the leak.