Investigate ByteBuf resource leak Azure-KeyValut

[blacelle]

Steps to reproduce:

Set up KeyVault Client with azure-security-keyvault-secrets: 4.1.0 and override Netty from 0.9.0 to 0.9.4 (as done by SpringBoot dependencies management) Fetch a few entries from KeyVault

2020-01-24 15:41:04,606 [      ] [reactor-http-nio-1] ERROR io.netty.util.ResourceLeakDetector.reportTracedLeak(320) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2134)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1319)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1231)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1268)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)

[blacelle]

https://github.com/Azure/azure-sdk-for-java/issues/6096

[joshfree]

/cc @g2vinay as fyi

[joshfree]

@anuchandy @alzimmermsft can you please follow up and ensure there's an external issue filed against 0.9.4?

[allenhumphreys]

I believe we're experiencing the same issue after updating to the latest version of the KeyVault library, specifically we upgraded to com.microsoft.azure:azure-spring-boot:2.2.4 which utilizes com.azure:azure-security-keyvault-secrets:4.1.1

I tried several methods to work around this issue such as up/downgrading reactor netty, as well as the netty core azure library. Nothing seems to get around it.

I'm including information around our setup to hopefully expedite a fix.

Is there any update on the progress of finding a fix?

Netty Leak Detection Logs

``` --2020-04-21 11:05:51.046 -ERROR -- [or-http-nio-2] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: #1: Hint: 'reactor.right.reactiveBridge' will handle the message from this point. io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86) io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88) io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28) io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) Created at: io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143) io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67) io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:292) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) --2020-04-21 11:05:52.252 -ERROR -- [or-http-nio-2] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: #1: io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #2: io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:292) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #3: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:872) io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:572) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:214) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #4: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:872) io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:552) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:214) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #5: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:872) io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:928) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:195) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #6: Hint: 'reactor.left.httpCodec' will handle the message from this point. io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #7: io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736) io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1496) io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166) io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #8: io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706) io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1496) io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166) io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2151) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1335) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) : 15 leak records were discarded because they were duplicates --2020-04-21 11:05:53.974 -ERROR -- [or-http-nio-2] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: #1: Hint: [id: 0x032065d6, L:/192.168.1.5:65042 - R:.vault.azure.net/:443] Buffered ByteBufHolder in Inbound Flux Queue io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86) io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88) io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28) reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:348) reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:352) reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:598) reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:90) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #2: Hint: 'reactor.right.reactiveBridge' will handle the message from this point. io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86) io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88) io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28) io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) Created at: io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143) io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67) io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:292) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) ```

I highly recommend running with the paranoid leak detection mode enabled via JVM arguments:

-Dio.netty.leakDetection.level=paranoid -Dio.netty.leakDetection.targetRecords=100

Our Setup:

• OS: macOS locally, Linux
• IDE : IntelliJ
  • JVM:

    openjdk version "11.0.5" 2019-10-15
    OpenJDK Runtime Environment (build 11.0.5+10)
    OpenJDK 64-Bit Server VM (build 11.0.5+10, mixed mode)

Gradle Dependencies Report

``` +--- :-common:1.0.457 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 | | | +--- org.jetbrains.kotlin:kotlin-stdlib-common:1.3.72 | | | \--- org.jetbrains:annotations:13.0 | | \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.3.70 -> 1.3.72 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-reactor:1.3.5 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*) | | +--- io.projectreactor:reactor-core:3.2.5.RELEASE -> 3.3.4.RELEASE | | | \--- org.reactivestreams:reactive-streams:1.0.3 | | \--- org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.3.5 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*) | | \--- org.reactivestreams:reactive-streams:1.0.2 -> 1.0.3 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-slf4j:1.3.5 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*) | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-rx2:1.3.5 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.3.5 (*) | | \--- io.reactivex.rxjava2:rxjava:2.2.8 -> 2.2.19 | | \--- org.reactivestreams:reactive-streams:1.0.3 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-jdk8:1.3.5 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*) | | \--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*) | +--- com.fasterxml.jackson.module:jackson-module-kotlin:2.10.3 | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3 | | | \--- com.fasterxml.jackson.core:jackson-core:2.10.3 | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3 | | \--- org.jetbrains.kotlin:kotlin-reflect:1.3.61 -> 1.3.72 | | \--- org.jetbrains.kotlin:kotlin-stdlib:1.3.72 (*) | +--- com.microsoft.azure:azure-servicebus-spring-boot-starter:2.2.4 | | +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.6.RELEASE | | | +--- org.springframework.boot:spring-boot:2.2.6.RELEASE | | | | +--- org.springframework:spring-core:5.2.5.RELEASE | | | | | \--- org.springframework:spring-jcl:5.2.5.RELEASE | | | | \--- org.springframework:spring-context:5.2.5.RELEASE | | | | +--- org.springframework:spring-aop:5.2.5.RELEASE | | | | | +--- org.springframework:spring-beans:5.2.5.RELEASE | | | | | | \--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | | | \--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | | +--- org.springframework:spring-beans:5.2.5.RELEASE (*) | | | | +--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | | \--- org.springframework:spring-expression:5.2.5.RELEASE | | | | \--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.6.RELEASE | | | | \--- org.springframework.boot:spring-boot:2.2.6.RELEASE (*) | | | +--- org.springframework.boot:spring-boot-starter-logging:2.2.6.RELEASE | | | | +--- ch.qos.logback:logback-classic:1.2.3 | | | | | +--- ch.qos.logback:logback-core:1.2.3 | | | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | +--- org.apache.logging.log4j:log4j-to-slf4j:2.12.1 | | | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | | \--- org.apache.logging.log4j:log4j-api:2.12.1 | | | | \--- org.slf4j:jul-to-slf4j:1.7.30 | | | | \--- org.slf4j:slf4j-api:1.7.30 | | | +--- jakarta.annotation:jakarta.annotation-api:1.3.5 | | | +--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | \--- org.yaml:snakeyaml:1.25 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.6.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*) | | | +--- jakarta.validation:jakarta.validation-api:2.0.2 | | | +--- org.apache.tomcat.embed:tomcat-embed-el:9.0.33 | | | \--- org.hibernate.validator:hibernate-validator:6.0.18.Final | | | +--- org.jboss.logging:jboss-logging:3.3.2.Final -> 3.4.1.Final | | | \--- com.fasterxml:classmate:1.3.4 -> 1.5.1 | | +--- com.microsoft.azure:azure-spring-boot:2.2.4 | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.0.RELEASE -> 2.2.6.RELEASE (*) | | | +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30 | | | +--- org.springframework:spring-web:5.2.0.RELEASE -> 5.2.5.RELEASE | | | | +--- org.springframework:spring-beans:5.2.5.RELEASE (*) | | | | \--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.0 -> 2.10.3 (*) | | | +--- com.azure:azure-security-keyvault-secrets:4.1.1 | | | | +--- com.azure:azure-core:1.3.0 | | | | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1 -> 2.10.3 | | | | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3 | | | | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.3 | | | | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*) | | | | | +--- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.10.1 -> 2.10.3 | | | | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.3 | | | | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3 | | | | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*) | | | | | | +--- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.10.3 | | | | | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3 | | | | | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.3 | | | | | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*) | | | | | | | +--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.2 -> 2.3.3 | | | | | | | | \--- jakarta.activation:jakarta.activation-api:1.2.2 | | | | | | | \--- jakarta.activation:jakarta.activation-api:1.2.1 -> 1.2.2 | | | | | | +--- org.codehaus.woodstox:stax2-api:4.2 | | | | | | \--- com.fasterxml.woodstox:woodstox-core:6.1.1 | | | | | | \--- org.codehaus.woodstox:stax2-api:4.2 | | | | | +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30 | | | | | +--- io.projectreactor:reactor-core:3.3.0.RELEASE -> 3.3.4.RELEASE (*) | | | | | \--- io.netty:netty-tcnative-boringssl-static:2.0.26.Final -> 2.0.30.Final | | | | \--- com.azure:azure-core-http-netty:1.4.0 | | | | +--- com.azure:azure-core:1.3.0 (*) | | | | +--- io.netty:netty-handler:4.1.44.Final -> 4.1.48.Final | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-resolver:4.1.48.Final | | | | | | \--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-buffer:4.1.48.Final | | | | | | \--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-transport:4.1.48.Final | | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | | +--- io.netty:netty-buffer:4.1.48.Final (*) | | | | | | \--- io.netty:netty-resolver:4.1.48.Final (*) | | | | | \--- io.netty:netty-codec:4.1.48.Final | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-buffer:4.1.48.Final (*) | | | | | \--- io.netty:netty-transport:4.1.48.Final (*) | | | | +--- io.netty:netty-handler-proxy:4.1.44.Final -> 4.1.48.Final | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-buffer:4.1.48.Final (*) | | | | | +--- io.netty:netty-transport:4.1.48.Final (*) | | | | | +--- io.netty:netty-codec:4.1.48.Final (*) | | | | | +--- io.netty:netty-codec-socks:4.1.48.Final | | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | | +--- io.netty:netty-buffer:4.1.48.Final (*) | | | | | | +--- io.netty:netty-transport:4.1.48.Final (*) | | | | | | \--- io.netty:netty-codec:4.1.48.Final (*) | | | | | \--- io.netty:netty-codec-http:4.1.48.Final | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-buffer:4.1.48.Final (*) | | | | | +--- io.netty:netty-transport:4.1.48.Final (*) | | | | | +--- io.netty:netty-codec:4.1.48.Final (*) | | | | | \--- io.netty:netty-handler:4.1.48.Final (*) | | | | +--- io.netty:netty-buffer:4.1.44.Final -> 4.1.48.Final (*) | | | | +--- io.netty:netty-codec-http:4.1.44.Final -> 4.1.48.Final (*) | | | | +--- io.netty:netty-codec-http2:4.1.44.Final -> 4.1.48.Final | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-buffer:4.1.48.Final (*) | | | | | +--- io.netty:netty-transport:4.1.48.Final (*) | | | | | +--- io.netty:netty-codec:4.1.48.Final (*) | | | | | +--- io.netty:netty-handler:4.1.48.Final (*) | | | | | \--- io.netty:netty-codec-http:4.1.48.Final (*) | | | | +--- io.netty:netty-transport-native-unix-common:4.1.44.Final -> 4.1.48.Final | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-buffer:4.1.48.Final (*) | | | | | \--- io.netty:netty-transport:4.1.48.Final (*) | | | | +--- io.netty:netty-transport-native-epoll:4.1.44.Final -> 4.1.48.Final | | | | | +--- io.netty:netty-common:4.1.48.Final | | | | | +--- io.netty:netty-buffer:4.1.48.Final (*) | | | | | +--- io.netty:netty-transport:4.1.48.Final (*) | | | | | \--- io.netty:netty-transport-native-unix-common:4.1.48.Final (*) | | | | \--- io.projectreactor.netty:reactor-netty:0.9.4.RELEASE -> 0.9.6.RELEASE | | | | +--- io.netty:netty-codec-http:4.1.48.Final (*) | | | | +--- io.netty:netty-codec-http2:4.1.48.Final (*) | | | | +--- io.netty:netty-handler:4.1.48.Final (*) | | | | +--- io.netty:netty-handler-proxy:4.1.48.Final (*) | | | | +--- io.netty:netty-transport-native-epoll:4.1.48.Final (*) | | | | \--- io.projectreactor:reactor-core:3.3.4.RELEASE (*) | | | \--- com.azure:azure-identity:1.0.4 | | | +--- com.azure:azure-core:1.3.0 (*) | | | +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30 | | | +--- com.microsoft.azure:msal4j:0.5.0-preview | | | | +--- com.nimbusds:oauth2-oidc-sdk:6.5 -> 6.14 | | | | | +--- com.sun.mail:javax.mail:1.6.1 | | | | | | \--- javax.activation:activation:1.1 | | | | | +--- com.github.stephenc.jcip:jcip-annotations:1.0-1 | | | | | +--- net.minidev:json-smart:[1.3.1,2.3] -> 2.3 | | | | | | \--- net.minidev:accessors-smart:1.2 | | | | | | \--- org.ow2.asm:asm:5.0.4 | | | | | +--- com.nimbusds:lang-tag:[1.4.3,) -> 1.4.4 | | | | | \--- com.nimbusds:nimbus-jose-jwt:[6.0.1,) -> 8.15 | | | | | +--- com.github.stephenc.jcip:jcip-annotations:1.0-1 | | | | | \--- net.minidev:json-smart:[1.3.1,2.3] -> 2.3 (*) | | | | +--- com.google.code.gson:gson:2.8.5 -> 2.8.6 | | | | +--- org.slf4j:slf4j-api:1.7.21 -> 1.7.30 | | | | +--- commons-codec:commons-codec:1.11 -> 1.13 | | | | +--- org.apache.commons:commons-lang3:3.9 | | | | +--- org.projectlombok:lombok:1.18.6 -> 1.18.12 | | | | \--- org.apache.httpcomponents:httpclient:4.5 -> 4.5.12 | | | | +--- org.apache.httpcomponents:httpcore:4.4.13 | | | | +--- commons-logging:commons-logging:1.2 | | | | \--- commons-codec:commons-codec:1.11 -> 1.13 | | | +--- com.nimbusds:oauth2-oidc-sdk:6.14 (*) | | | +--- org.nanohttpd:nanohttpd:2.3.1 | | | \--- net.java.dev.jna:jna-platform:5.4.0 -> 4.5.2 | | | \--- net.java.dev.jna:jna:4.5.2 | | +--- com.microsoft.azure:azure-servicebus:1.2.15 -> 1.2.18 | | | +--- org.apache.qpid:proton-j:0.31.0 | | | +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.30 | | | \--- com.microsoft.azure:adal4j:1.3.0 -> 1.6.4 | | | +--- com.nimbusds:oauth2-oidc-sdk:6.5 -> 6.14 (*) | | | +--- com.google.code.gson:gson:2.8.0 -> 2.8.6 | | | +--- org.slf4j:slf4j-api:1.7.21 -> 1.7.30 | | | +--- commons-codec:commons-codec:1.11 -> 1.13 | | | \--- org.apache.commons:commons-lang3:3.5 -> 3.9 | | +--- com.google.code.findbugs:annotations:2.0.1 | | +--- com.fasterxml.jackson.core:jackson-core:2.10.0 -> 2.10.3 | | \--- commons-codec:commons-codec:1.13 | +--- com.microsoft.azure:azure-servicebus:1.2.18 (*) | +--- com.microsoft.azure:applicationinsights-logging-logback:2.6.0 | | +--- com.microsoft.azure:applicationinsights-core:2.6.0 | | +--- ch.qos.logback:logback-classic:1.2.3 (*) | | \--- ch.qos.logback:logback-core:1.2.3 | +--- org.springframework.boot:spring-boot-starter-security:2.2.6.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*) | | +--- org.springframework:spring-aop:5.2.5.RELEASE (*) | | +--- org.springframework.security:spring-security-config:5.2.2.RELEASE | | | +--- org.springframework.security:spring-security-core:5.2.2.RELEASE | | | | +--- org.springframework:spring-aop:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | | +--- org.springframework:spring-beans:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | | +--- org.springframework:spring-context:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | | +--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | | \--- org.springframework:spring-expression:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | +--- org.springframework:spring-aop:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | +--- org.springframework:spring-beans:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | +--- org.springframework:spring-context:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | \--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | \--- org.springframework.security:spring-security-web:5.2.2.RELEASE | | +--- org.springframework.security:spring-security-core:5.2.2.RELEASE (*) | | +--- org.springframework:spring-aop:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | +--- org.springframework:spring-beans:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | +--- org.springframework:spring-context:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | +--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | +--- org.springframework:spring-expression:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | \--- org.springframework:spring-web:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | +--- io.projectreactor:reactor-core:3.3.4.RELEASE (*) | +--- io.projectreactor.addons:reactor-adapter:3.3.3.RELEASE | | \--- io.projectreactor:reactor-core:3.3.4.RELEASE (*) | +--- io.projectreactor.addons:reactor-extra:3.3.3.RELEASE | | \--- io.projectreactor:reactor-core:3.3.4.RELEASE (*) | +--- io.projectreactor.kotlin:reactor-kotlin-extensions:1.0.2.RELEASE | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.4.RELEASE (*) | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.72 (*) | +--- org.jetbrains.kotlin:kotlin-reflect:1.3.72 (*) | +--- net.minidev:json-smart:2.3 (*) | +--- com.microsoft.azure:azure-eventgrid:1.3.0 | | \--- com.microsoft.azure:azure-client-runtime:1.5.0 -> 1.6.13 | | \--- com.microsoft.rest:client-runtime:1.6.13 | | +--- com.google.guava:guava:20.0 -> 28.2-jre | | | +--- com.google.guava:failureaccess:1.0.1 | | | +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava | | | +--- com.google.code.findbugs:jsr305:3.0.2 | | | +--- org.checkerframework:checker-qual:2.10.0 | | | +--- com.google.errorprone:error_prone_annotations:2.3.4 | | | \--- com.google.j2objc:j2objc-annotations:1.3 | | +--- com.squareup.retrofit2:retrofit:2.5.0 | | | \--- com.squareup.okhttp3:okhttp:3.12.0 -> 3.12.2 | | | \--- com.squareup.okio:okio:1.15.0 | | +--- com.squareup.okhttp3:okhttp:3.12.2 (*) | | +--- com.squareup.okhttp3:logging-interceptor:3.12.2 -> 3.12.0 | | | \--- com.squareup.okhttp3:okhttp:3.12.0 -> 3.12.2 (*) | | +--- com.squareup.okhttp3:okhttp-urlconnection:3.12.2 -> 3.14.7 | | | \--- com.squareup.okhttp3:okhttp:3.14.7 -> 3.12.2 (*) | | +--- com.squareup.retrofit2:converter-jackson:2.5.0 | | | +--- com.squareup.retrofit2:retrofit:2.5.0 (*) | | | \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.10.3 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-joda:2.9.8 -> 2.10.3 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.3 | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*) | | | \--- joda-time:joda-time:2.9.9 -> 2.10.5 | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | +--- io.reactivex:rxjava:1.3.8 | | +--- com.squareup.retrofit2:adapter-rxjava:2.4.0 | | | +--- com.squareup.retrofit2:retrofit:2.4.0 -> 2.5.0 (*) | | | \--- io.reactivex:rxjava:1.3.0 -> 1.3.8 | | +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.30 | | \--- com.microsoft.azure:azure-annotations:1.7.0 | +--- com.microsoft.azure:azure-cosmos:3.7.2 | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.1 -> 2.10.3 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1 -> 2.10.3 (*) | | +--- com.fasterxml.uuid:java-uuid-generator:3.2.0 | | +--- commons-io:commons-io:2.5 | | +--- commons-validator:commons-validator:1.6 | | | +--- commons-digester:commons-digester:1.8.1 | | | +--- commons-logging:commons-logging:1.2 | | | \--- commons-collections:commons-collections:3.2.2 | | +--- io.projectreactor:reactor-core:3.3.0.RELEASE -> 3.3.4.RELEASE (*) | | +--- io.netty:netty-codec-http:4.1.44.Final -> 4.1.48.Final (*) | | +--- io.netty:netty-handler:4.1.44.Final -> 4.1.48.Final (*) | | +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30 | | +--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | +--- org.apache.commons:commons-collections4:4.2 | | +--- org.apache.commons:commons-text:1.6 | | | \--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | +--- com.google.guava:guava:25.0-jre -> 28.2-jre (*) | | +--- io.dropwizard.metrics:metrics-core:4.1.0 -> 4.1.5 | | | \--- org.slf4j:slf4j-api:1.7.30 | | +--- io.micrometer:micrometer-core:1.2.0 -> 1.3.6 | | | +--- org.hdrhistogram:HdrHistogram:2.1.11 | | | \--- org.latencyutils:LatencyUtils:2.0.3 | | +--- io.reactivex.rxjava2:rxjava:2.2.4 -> 2.2.19 (*) | | \--- io.projectreactor.netty:reactor-netty:0.9.4.RELEASE -> 0.9.6.RELEASE (*) | +--- com.microsoft.azure:azure-cosmosdb-spring-boot-starter:2.2.4 | | +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.6.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.6.RELEASE (*) | | +--- com.microsoft.azure:azure-spring-boot:2.2.4 (*) | | \--- com.microsoft.azure:spring-data-cosmosdb:2.2.3.FIX1 | | +--- org.springframework:spring-core:5.2.0.RELEASE -> 5.2.5.RELEASE (*) | | +--- org.springframework:spring-web:5.2.0.RELEASE -> 5.2.5.RELEASE (*) | | +--- org.springframework:spring-beans:5.2.0.RELEASE -> 5.2.5.RELEASE (*) | | +--- org.springframework:spring-context:5.2.0.RELEASE -> 5.2.5.RELEASE (*) | | +--- org.springframework:spring-tx:5.2.0.RELEASE -> 5.2.5.RELEASE | | | +--- org.springframework:spring-beans:5.2.5.RELEASE (*) | | | \--- org.springframework:spring-core:5.2.5.RELEASE (*) | | +--- org.springframework.data:spring-data-commons:2.2.0.RELEASE -> 2.2.6.RELEASE | | | +--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | +--- org.springframework:spring-beans:5.2.5.RELEASE (*) | | | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | | +--- org.springframework:spring-expression:5.2.0.RELEASE -> 5.2.5.RELEASE (*) | | +--- com.microsoft.azure:azure-cosmos:3.7.1 -> 3.7.2 (*) | | +--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.10.0 -> 2.10.3 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.3 | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.10.0 -> 2.10.3 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.3 | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.0 -> 2.10.3 (*) | | +--- org.json:json:20140107 | | \--- org.javatuples:javatuples:1.2 | +--- com.microsoft.azure:applicationinsights-spring-boot-starter:2.6.0 | | \--- com.microsoft.azure:applicationinsights-web:2.6.0 | +--- org.springframework.boot:spring-boot-starter-actuator:2.2.6.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*) | | +--- org.springframework.boot:spring-boot-actuator-autoconfigure:2.2.6.RELEASE | | | +--- org.springframework.boot:spring-boot-actuator:2.2.6.RELEASE | | | | +--- org.springframework.boot:spring-boot:2.2.6.RELEASE (*) | | | | \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.3 (*) | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.6.RELEASE (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*) | | | +--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | +--- org.springframework:spring-context:5.2.5.RELEASE (*) | | | \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.3 (*) | | \--- io.micrometer:micrometer-core:1.3.6 (*) | +--- org.springframework.boot:spring-boot-starter-webflux:2.2.6.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-json:2.2.6.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*) | | | +--- org.springframework:spring-web:5.2.5.RELEASE (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*) | | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.10.3 (*) | | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.3 (*) | | | \--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.10.3 (*) | | +--- org.springframework.boot:spring-boot-starter-reactor-netty:2.2.6.RELEASE | | | +--- io.projectreactor.netty:reactor-netty:0.9.6.RELEASE (*) | | | \--- org.glassfish:jakarta.el:3.0.3 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.6.RELEASE (*) | | +--- org.springframework:spring-web:5.2.5.RELEASE (*) | | +--- org.springframework:spring-webflux:5.2.5.RELEASE | | | +--- org.springframework:spring-beans:5.2.5.RELEASE (*) | | | +--- org.springframework:spring-core:5.2.5.RELEASE (*) | | | +--- org.springframework:spring-web:5.2.5.RELEASE (*) | | | \--- io.projectreactor:reactor-core:3.3.4.RELEASE (*) | | \--- org.synchronoss.cloud:nio-multipart-parser:1.1.0 | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | \--- org.synchronoss.cloud:nio-stream-storage:1.1.3 | | \--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | +--- io.springfox:springfox-swagger2:3.0.0-SNAPSHOT | | +--- io.swagger:swagger-annotations:1.5.20 | | +--- io.swagger:swagger-models:1.5.20 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.5 -> 2.10.3 | | | +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.30 | | | \--- io.swagger:swagger-annotations:1.5.20 | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT | | | \--- io.springfox:springfox-core:3.0.0-SNAPSHOT | | | +--- net.bytebuddy:byte-buddy:1.9.10 -> 1.10.8 | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE | | | | +--- org.springframework:spring-beans:5.2.0.RELEASE -> 5.2.5.RELEASE (*) | | | | +--- org.springframework:spring-context:5.2.0.RELEASE -> 5.2.5.RELEASE (*) | | | | +--- org.springframework:spring-aop:5.2.0.RELEASE -> 5.2.5.RELEASE (*) | | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | \--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- io.springfox:springfox-schema:3.0.0-SNAPSHOT | | | +--- io.springfox:springfox-core:3.0.0-SNAPSHOT (*) | | | \--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | +--- io.springfox:springfox-swagger-common:3.0.0-SNAPSHOT | | | +--- io.swagger:swagger-annotations:1.5.20 | | | +--- io.swagger:swagger-models:1.5.20 (*) | | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | | +--- io.springfox:springfox-schema:3.0.0-SNAPSHOT (*) | | | +--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT | | | | +--- io.github.classgraph:classgraph:4.1.7 | | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | | | \--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | \--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | +--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT (*) | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | \--- org.mapstruct:mapstruct:1.3.1.Final | +--- io.springfox:springfox-spring-webflux:3.0.0-SNAPSHOT | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | \--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT (*) | +--- org.springframework.security:spring-security-oauth2-jose:5.3.1.RELEASE -> 5.2.2.RELEASE | | +--- com.nimbusds:nimbus-jose-jwt:7.8.1 -> 8.15 (*) | | +--- org.springframework.security:spring-security-core:5.2.2.RELEASE (*) | | +--- org.springframework.security:spring-security-oauth2-core:5.2.2.RELEASE | | | +--- org.springframework.security:spring-security-core:5.2.2.RELEASE (*) | | | +--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | | \--- org.springframework:spring-web:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | | \--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*) | +--- com.nimbusds:nimbus-jose-jwt:8.15 (*) | +--- com.onelogin:java-saml:2.5.0 | | +--- com.onelogin:java-saml-core:2.5.0 | | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | | +--- joda-time:joda-time:2.10.3 -> 2.10.5 | | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | | +--- org.apache.santuario:xmlsec:2.1.4 | | | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | | | | +--- com.fasterxml.woodstox:woodstox-core:5.0.3 -> 6.1.1 (*) | | | | \--- commons-codec:commons-codec:1.12 -> 1.13 | | | \--- commons-codec:commons-codec:1.12 -> 1.13 | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | +--- joda-time:joda-time:2.10.3 -> 2.10.5 | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | +--- org.apache.santuario:xmlsec:2.1.4 (*) | | \--- commons-codec:commons-codec:1.12 -> 1.13 | \--- io.reactivex:rxjava-reactive-streams:1.2.1 | +--- io.reactivex:rxjava:1.2.2 -> 1.3.8 | \--- org.reactivestreams:reactive-streams:1.0.0 -> 1.0.3 +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.72 (*) +--- org.jetbrains.kotlin:kotlin-reflect:1.3.72 (*) +--- org.springframework.boot:spring-boot-starter-actuator:2.2.6.RELEASE (*) +--- org.springframework.boot:spring-boot-starter-webflux:2.2.6.RELEASE (*) +--- org.springframework.cloud:spring-cloud-starter-config:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-starter:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.5.RELEASE -> 2.2.6.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE | | | \--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE | | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE | | | \--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE | | \--- org.springframework.security:spring-security-rsa:1.0.9.RELEASE | | \--- org.bouncycastle:bcpkix-jdk15on:1.64 | | \--- org.bouncycastle:bcprov-jdk15on:1.64 | +--- org.springframework.cloud:spring-cloud-config-client:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.5.RELEASE -> 2.2.6.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE (*) | | +--- org.springframework:spring-web:5.2.4.RELEASE -> 5.2.5.RELEASE (*) | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 -> 2.10.3 | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 -> 2.10.3 (*) | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 -> 2.10.3 (*) +--- org.springframework.cloud:spring-cloud-config-server:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-config-client:2.2.2.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-actuator:2.2.5.RELEASE -> 2.2.6.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-web:2.2.5.RELEASE -> 2.2.6.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-json:2.2.6.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-tomcat:2.2.6.RELEASE | | | +--- jakarta.annotation:jakarta.annotation-api:1.3.5 | | | +--- org.apache.tomcat.embed:tomcat-embed-core:9.0.33 | | | +--- org.apache.tomcat.embed:tomcat-embed-el:9.0.33 | | | \--- org.apache.tomcat.embed:tomcat-embed-websocket:9.0.33 | | | \--- org.apache.tomcat.embed:tomcat-embed-core:9.0.33 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.6.RELEASE (*) | | +--- org.springframework:spring-web:5.2.5.RELEASE (*) | | \--- org.springframework:spring-webmvc:5.2.5.RELEASE | | +--- org.springframework:spring-aop:5.2.5.RELEASE (*) | | +--- org.springframework:spring-beans:5.2.5.RELEASE (*) | | +--- org.springframework:spring-context:5.2.5.RELEASE (*) | | +--- org.springframework:spring-core:5.2.5.RELEASE (*) | | +--- org.springframework:spring-expression:5.2.5.RELEASE (*) | | \--- org.springframework:spring-web:5.2.5.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-validation:2.2.5.RELEASE -> 2.2.6.RELEASE (*) | +--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE | +--- org.springframework.security:spring-security-rsa:1.0.9.RELEASE (*) | +--- org.eclipse.jgit:org.eclipse.jgit:5.1.3.201810200350-r | | +--- com.jcraft:jsch:0.1.54 | | +--- com.jcraft:jzlib:1.1.1 | | +--- com.googlecode.javaewah:JavaEWAH:1.1.6 | | \--- org.slf4j:slf4j-api:1.7.2 -> 1.7.30 | +--- org.eclipse.jgit:org.eclipse.jgit.http.apache:5.1.3.201810200350-r | | +--- org.eclipse.jgit:org.eclipse.jgit:5.1.3.201810200350-r (*) | | +--- org.apache.httpcomponents:httpclient:4.5.5 -> 4.5.12 (*) | | \--- org.apache.httpcomponents:httpcore:4.4.9 -> 4.4.13 | +--- org.yaml:snakeyaml:1.25 | \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.2 -> 2.10.3 | +--- org.yaml:snakeyaml:1.24 -> 1.25 | \--- com.fasterxml.jackson.core:jackson-core:2.10.3 +--- org.springframework.cloud:spring-cloud-starter-openfeign:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-starter:2.2.2.RELEASE (*) | +--- org.springframework.cloud:spring-cloud-openfeign-core:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.5.RELEASE -> 2.2.6.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-netflix-ribbon:2.2.2.RELEASE | | | \--- org.springframework.cloud:spring-cloud-netflix-archaius:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-starter-aop:2.2.5.RELEASE -> 2.2.6.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*) | | | +--- org.springframework:spring-aop:5.2.5.RELEASE (*) | | | \--- org.aspectj:aspectjweaver:1.9.5 | | \--- io.github.openfeign.form:feign-form-spring:3.8.0 | | +--- io.github.openfeign.form:feign-form:3.8.0 | | | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | | +--- org.springframework:spring-web:5.1.5.RELEASE -> 5.2.5.RELEASE (*) | | +--- commons-fileupload:commons-fileupload:1.4 | | | \--- commons-io:commons-io:2.2 -> 2.5 | | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | +--- org.springframework:spring-web:5.2.4.RELEASE -> 5.2.5.RELEASE (*) | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE (*) | +--- io.github.openfeign:feign-core:10.7.4 | +--- io.github.openfeign:feign-slf4j:10.7.4 | | +--- io.github.openfeign:feign-core:10.7.4 | | \--- org.slf4j:slf4j-api:1.7.13 -> 1.7.30 | \--- io.github.openfeign:feign-hystrix:10.7.4 | +--- io.github.openfeign:feign-core:10.7.4 | +--- com.netflix.archaius:archaius-core:0.7.6 | | +--- com.google.code.findbugs:jsr305:3.0.1 -> 3.0.2 | | +--- commons-configuration:commons-configuration:1.8 | | | +--- commons-lang:commons-lang:2.6 | | | \--- commons-logging:commons-logging:1.1.1 -> 1.2 | | +--- org.slf4j:slf4j-api:1.6.4 -> 1.7.30 | | +--- com.google.guava:guava:16.0 -> 28.2-jre (*) | | +--- com.fasterxml.jackson.core:jackson-annotations:2.4.3 -> 2.10.3 | | +--- com.fasterxml.jackson.core:jackson-core:2.4.3 -> 2.10.3 | | \--- com.fasterxml.jackson.core:jackson-databind:2.4.3 -> 2.10.3 (*) | \--- com.netflix.hystrix:hystrix-core:1.5.18 | +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.30 | +--- com.netflix.archaius:archaius-core:0.4.1 -> 0.7.6 (*) | +--- io.reactivex:rxjava:1.2.0 -> 1.3.8 | \--- org.hdrhistogram:HdrHistogram:2.1.9 -> 2.1.11 +--- org.springframework.data:spring-data-commons:2.2.6.RELEASE (*) +--- io.springfox:springfox-swagger2:3.0.0-SNAPSHOT (*) +--- io.springfox:springfox-spring-webflux:3.0.0-SNAPSHOT (*) +--- io.springfox:springfox-swagger-ui:3.0.0-SNAPSHOT +--- javax.persistence:persistence-api:1.0.2 +--- io.projectreactor:reactor-core:3.3.4.RELEASE (*) +--- com.github.fge:json-patch:1.9 | \--- com.github.fge:jackson-coreutils:1.6 | +--- com.github.fge:msg-simple:1.1 | | +--- com.github.fge:btf:1.2 | | | \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2 | | \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2 | +--- com.fasterxml.jackson.core:jackson-databind:2.2.3 -> 2.10.3 (*) | +--- com.google.guava:guava:16.0.1 -> 28.2-jre (*) | \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2 +--- com.microsoft.azure:azure-cosmosdb-spring-boot-starter:2.2.4 (*) +--- com.microsoft.azure:azure-keyvault-secrets-spring-boot-starter:2.2.4 | +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.6.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.6.RELEASE (*) | +--- com.microsoft.azure:azure-spring-boot:2.2.4 (*) | +--- com.microsoft.azure:azure-client-authentication:1.6.13 | | +--- com.microsoft.azure:azure-client-runtime:1.6.13 (*) | | +--- commons-codec:commons-codec:1.11 -> 1.13 | | +--- com.microsoft.azure:adal4j:1.6.4 (*) | | \--- com.microsoft.azure:azure-annotations:1.7.0 | \--- commons-io:commons-io:2.3 -> 2.5 +--- com.microsoft.azure:azure-cosmos:3.7.2 (*) +--- com.microsoft.azure:applicationinsights-spring-boot-starter:2.6.0 (*) +--- com.squareup.okhttp3:okhttp:3.12.2 (*) +--- com.google.guava:guava:28.2-jre (*) +--- org.jsmpp:jsmpp:2.3.7 | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 +--- org.springframework.boot:spring-boot-starter-cache:2.2.6.RELEASE | +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*) | \--- org.springframework:spring-context-support:5.2.5.RELEASE | +--- org.springframework:spring-beans:5.2.5.RELEASE (*) | +--- org.springframework:spring-context:5.2.5.RELEASE (*) | \--- org.springframework:spring-core:5.2.5.RELEASE (*) +--- javax.cache:cache-api:1.1.1 \--- org.ehcache:ehcache:3.8.1 +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 \--- org.glassfish.jaxb:jaxb-runtime:2.3.1 -> 2.3.2 +--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.2 -> 2.3.3 (*) +--- org.glassfish.jaxb:txw2:2.3.2 +--- com.sun.istack:istack-commons-runtime:3.0.8 | \--- jakarta.activation:jakarta.activation-api:1.2.1 -> 1.2.2 +--- org.jvnet.staxex:stax-ex:1.8.1 | +--- jakarta.activation:jakarta.activation-api:1.2.1 -> 1.2.2 | \--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.2 -> 2.3.3 (*) +--- com.sun.xml.fastinfoset:FastInfoset:1.2.16 \--- jakarta.activation:jakarta.activation-api:1.2.1 -> 1.2.2 ```

[anuchandy]

@allenhumphreys, @blacelle - I can repro this using the dependencies in the maven tree. Looking into this.

[anuchandy]

The above PR fixing this has been merged. \\cc @allenhumphreys, @blacelle

[timwhit]

@anuchandy any idea when a new version will be released?

[anuchandy]

@timwhit - we have a patch release planned around mid of May. I'll update this thread once bits are out.

[blacelle]

@anuchandy Please confirm the library/version supposed to fix this.

[anuchandy]

The version "com.azure:azure-security-keyvault-secrets:4.1.3" include the fix.

[tysonnorris]

We are seeing the same error with some of our tests that exercise cosmosdb sdk, using com.azure:azure-cosmos:4.0.1 with jdk 1.8 and jdk 11. I'm trying to narrow down a test case, but it happens intermittently, about 50% of our test runs. Let me know if you have any advice of possible fix for cosmos sdk apis?

[anuchandy]

@tysonnorris thanks for reaching out. Can you open a new issue targetting cosmos-SDK with the findings so far from your side?

[anuchandy]

@tysonnorris I just want to provide bit more details on reason I requested a new issue.

The original problem specifically was due to key-vault library not closing stream in the pipeline.

The ByteBuf leak could happen if the stream is not closed in the library or in the user app. We need to investigate the issue you are seeing in the context of cosmos-sdk, I'm opening a new issue and will tag you in it.

Sorry that this issue's title is misleading, I just corrected it to avoid any confusion in the future.

[krishgeeneuron]

Hi,

We are facing the same issue with azure keyvault 4.1.5. Please could someone help us identify the right version the below leak issue is fixed. Is there any azure key vault version below 4.2.0 which has the fix ?

{ "timestamp": 1625580889173, "status": 500, "error": "Internal Server Error", "message": "Request processing failed; nested exception is java.lang.RuntimeException: Max retries 3 times exceeded. Error Details: failed to allocate 16777216 byte(s) of direct memory (used: 486539271, max: 501743616)" }