search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.34k stars 1.98k forks source link

[BUG] Cosmos SDK Netty ByteBuf leak #9023

Closed allenhumphreys closed 4 years ago

allenhumphreys commented 4 years ago

Describe the bug There is a leak in the Cosmos SDK when using the DIRECT connection mode.

The problem seems to only become obvious when services are under continuous medium load. We've had Cosmos deployed for many months and this hasn't been an issue, but shortly after going live and having sustained traffic increase above a certain level, all our services started running out of heap memory and experiencing OutOfMemory exceptions.

A leak this bad is highly concerning. Fortunately we were able to switch our service to use GATEWAY mode to bypass the issue.

We went through several mitigation steps, including:

All to have it become abundantly clear that the leak was coming from the Cosmos SDK, and that there was nothing we could do.

We subsequently enabled ADVANCED Netty leak detection to get a record of the last code to access the ByteBuf. That log showed that the last access is a call to release() at RntbdResponse.java:198. That code has suspicious comments, but more importantly it helped me confirm that attempting to switch to GATEWAY connection mode would likely bypass the leaking behavior, which it did.

Netty Leak Detection Log

``` --2020-03-10 15:18:36.798 -ERROR -- [rntbd-nio-2-1] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: #1: com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.lambda$release$0(RntbdResponse.java:198) java.util.concurrent.atomic.AtomicInteger.accumulateAndGet(AtomicInteger.java:289) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.release(RntbdResponse.java:184) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.release(RntbdResponse.java:172) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdRequestManager.channelRead(RntbdRequestManager.java:196) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #2: com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdToken.getValue(RntbdToken.java:85) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdToken.getValue(RntbdToken.java:95) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.toIntegerEntry(RntbdResponseHeaders.java:496) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.lambda$collectEntries$44(RntbdResponseHeaders.java:469) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.lambda$asList$0(RntbdResponseHeaders.java:203) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.collectEntries(RntbdResponseHeaders.java:468) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.asList(RntbdResponseHeaders.java:201) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.toStoreResponse(RntbdResponse.java:239) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdRequestManager.messageReceived(RntbdRequestManager.java:707) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdRequestManager.channelRead(RntbdRequestManager.java:175) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #3: io.netty.buffer.AdvancedLeakAwareByteBuf.readByte(AdvancedLeakAwareByteBuf.java:400) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdTokenStream.decode(RntbdTokenStream.java:68) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.decode(RntbdResponseHeaders.java:227) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.decode(RntbdResponse.java:104) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseDecoder.decode(RntbdResponseDecoder.java:32) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #4: io.netty.buffer.AdvancedLeakAwareByteBuf.readShortLE(AdvancedLeakAwareByteBuf.java:826) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdTokenStream.decode(RntbdTokenStream.java:67) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.decode(RntbdResponseHeaders.java:227) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.decode(RntbdResponse.java:104) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseDecoder.decode(RntbdResponseDecoder.java:32) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #5: io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdTokenType$RntbdShortString.readSlice(RntbdTokenType.java:671) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdToken.decode(RntbdToken.java:156) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdTokenStream.decode(RntbdTokenStream.java:76) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.decode(RntbdResponseHeaders.java:227) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.decode(RntbdResponse.java:104) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseDecoder.decode(RntbdResponseDecoder.java:32) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #6: io.netty.buffer.AdvancedLeakAwareByteBuf.readSlice(AdvancedLeakAwareByteBuf.java:118) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdTokenType$RntbdByte.readSlice(RntbdTokenType.java:150) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdToken.decode(RntbdToken.java:156) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdTokenStream.decode(RntbdTokenStream.java:76) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseHeaders.decode(RntbdResponseHeaders.java:227) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.decode(RntbdResponse.java:104) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseDecoder.decode(RntbdResponseDecoder.java:32) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #7: io.netty.buffer.AdvancedLeakAwareByteBuf.readShortLE(AdvancedLeakAwareByteBuf.java:826) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdUUID.decode(RntbdUUID.java:52) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseStatus.decode(RntbdResponseStatus.java:80) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponse.decode(RntbdResponse.java:103) com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdResponseDecoder.decode(RntbdResponseDecoder.java:32) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #8: Hint: 'RntbdResponseDecoder#0' will handle the message from this point. io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #9: Hint: 'IdleStateHandler#0' will handle the message from this point. io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #10: io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736) io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1480) io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166) io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1324) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) #11: io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706) io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1480) io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166) io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1324) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) Created at: io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:332) io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:168) io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:159) io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117) io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2136) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1319) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:748) : 144 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit. ```

To Reproduce Steps to reproduce the behavior: Unfortunately, I have not been able to reproduce this in isolation, only in production, as it seems to require a certain amount of sustained traffic to happen.

Screenshots Heap usage during leaking behavior:

Screen Shot 2020-03-12 at 10 41 16 AM

Heap usage after switching to GATEWAY mode:

Screen Shot 2020-03-12 at 11 08 22 AM

Setup (please complete the following information):

We are running our services in AKS using open JDK 8

Gradle Runtime Dependency Report

``` +--- com.redacted.redacted:redacted-common:1.0.390 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.70 | | | +--- org.jetbrains.kotlin:kotlin-stdlib-common:1.3.70 | | | \--- org.jetbrains:annotations:13.0 | | \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.3.61 -> 1.3.70 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-reactor:1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.70 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | | +--- io.projectreactor:reactor-core:3.2.5.RELEASE -> 3.3.3.RELEASE | | | \--- org.reactivestreams:reactive-streams:1.0.3 | | \--- org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.70 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | | \--- org.reactivestreams:reactive-streams:1.0.2 -> 1.0.3 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-slf4j:1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.70 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-rx2:1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.70 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.3.3 (*) | | \--- io.reactivex.rxjava2:rxjava:2.2.8 -> 2.2.17 | | \--- org.reactivestreams:reactive-streams:1.0.3 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-jdk8:1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.70 (*) | | \--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | +--- com.fasterxml.jackson.module:jackson-module-kotlin:2.10.3 -> 2.10.2 | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | | \--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | \--- org.jetbrains.kotlin:kotlin-reflect:1.3.61 -> 1.3.70 | | \--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 (*) | +--- com.microsoft.azure:azure-servicebus-spring-boot-starter:2.2.1 | | +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot:2.2.4.RELEASE | | | | +--- org.springframework:spring-core:5.2.3.RELEASE | | | | | \--- org.springframework:spring-jcl:5.2.3.RELEASE | | | | \--- org.springframework:spring-context:5.2.3.RELEASE | | | | +--- org.springframework:spring-aop:5.2.3.RELEASE | | | | | +--- org.springframework:spring-beans:5.2.3.RELEASE | | | | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | | \--- org.springframework:spring-expression:5.2.3.RELEASE | | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.4.RELEASE | | | | \--- org.springframework.boot:spring-boot:2.2.4.RELEASE (*) | | | +--- org.springframework.boot:spring-boot-starter-logging:2.2.4.RELEASE | | | | +--- ch.qos.logback:logback-classic:1.2.3 | | | | | +--- ch.qos.logback:logback-core:1.2.3 | | | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | +--- org.apache.logging.log4j:log4j-to-slf4j:2.12.1 | | | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | | \--- org.apache.logging.log4j:log4j-api:2.12.1 | | | | \--- org.slf4j:jul-to-slf4j:1.7.30 | | | | \--- org.slf4j:slf4j-api:1.7.30 | | | +--- jakarta.annotation:jakarta.annotation-api:1.3.5 | | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | \--- org.yaml:snakeyaml:1.25 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | | +--- jakarta.validation:jakarta.validation-api:2.0.2 | | | +--- org.apache.tomcat.embed:tomcat-embed-el:9.0.30 | | | \--- org.hibernate.validator:hibernate-validator:6.0.18.Final | | | +--- org.jboss.logging:jboss-logging:3.3.2.Final -> 3.4.1.Final | | | \--- com.fasterxml:classmate:1.3.4 -> 1.5.1 | | +--- com.microsoft.azure:azure-spring-boot:2.2.1 | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.0.RELEASE -> 2.2.4.RELEASE (*) | | | +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30 | | | +--- org.springframework:spring-web:5.2.0.RELEASE -> 5.2.3.RELEASE | | | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.0 -> 2.10.2 (*) | | +--- com.microsoft.azure:azure-servicebus:1.2.15 -> 1.2.18 | | | +--- org.apache.qpid:proton-j:0.31.0 | | | +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.30 | | | \--- com.microsoft.azure:adal4j:1.3.0 | | | +--- com.nimbusds:oauth2-oidc-sdk:5.24.1 | | | | +--- javax.mail:mail:1.4.7 | | | | | \--- javax.activation:activation:1.1 | | | | +--- com.github.stephenc.jcip:jcip-annotations:1.0-1 | | | | +--- org.apache.commons:commons-lang3:3.5 -> 3.9 | | | | +--- org.apache.commons:commons-collections4:4.1 -> 4.2 | | | | +--- net.minidev:json-smart:1.3.1 -> 2.3 | | | | | \--- net.minidev:accessors-smart:1.2 | | | | | \--- org.ow2.asm:asm:5.0.4 | | | | +--- com.nimbusds:lang-tag:[1.4.3,) -> 1.4.4 | | | | \--- com.nimbusds:nimbus-jose-jwt:[4.29,) -> 8.9 | | | | +--- com.github.stephenc.jcip:jcip-annotations:1.0-1 | | | | \--- net.minidev:json-smart:[1.3.1,2.3] -> 2.3 (*) | | | +--- com.google.code.gson:gson:2.2.4 -> 2.8.6 | | | +--- org.slf4j:slf4j-api:1.7.5 -> 1.7.30 | | | \--- commons-codec:commons-codec:1.10 -> 1.13 | | +--- com.google.code.findbugs:annotations:2.0.1 | | +--- com.fasterxml.jackson.core:jackson-core:2.10.0 -> 2.10.2 | | \--- commons-codec:commons-codec:1.13 | +--- com.microsoft.azure:azure-servicebus:1.2.18 (*) | +--- com.microsoft.azure:applicationinsights-logging-logback:2.5.1 | | +--- com.microsoft.azure:applicationinsights-core:2.5.1 | | +--- ch.qos.logback:logback-classic:1.2.3 (*) | | \--- ch.qos.logback:logback-core:1.2.3 | +--- org.springframework.boot:spring-boot-starter-security:2.2.4.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | +--- org.springframework:spring-aop:5.2.3.RELEASE (*) | | +--- org.springframework.security:spring-security-config:5.2.1.RELEASE | | | +--- org.springframework.security:spring-security-core:5.2.1.RELEASE | | | | +--- org.springframework:spring-aop:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-beans:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-context:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | | \--- org.springframework:spring-expression:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | +--- org.springframework:spring-aop:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | +--- org.springframework:spring-beans:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | +--- org.springframework:spring-context:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | \--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | \--- org.springframework.security:spring-security-web:5.2.1.RELEASE | | +--- org.springframework.security:spring-security-core:5.2.1.RELEASE (*) | | +--- org.springframework:spring-aop:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-beans:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-context:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-expression:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | \--- org.springframework:spring-web:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | +--- io.projectreactor:reactor-core:3.3.3.RELEASE (*) | +--- io.projectreactor.addons:reactor-adapter:3.3.2.RELEASE | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | +--- io.projectreactor.addons:reactor-extra:3.3.2.RELEASE | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | +--- io.projectreactor.kotlin:reactor-kotlin-extensions:1.0.2.RELEASE | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 (*) | +--- org.jetbrains.kotlin:kotlin-reflect:1.3.70 (*) | +--- net.minidev:json-smart:2.3 (*) | +--- com.microsoft.azure:azure-eventgrid:1.3.0 | | \--- com.microsoft.azure:azure-client-runtime:1.5.0 -> 1.6.12 | | \--- com.microsoft.rest:client-runtime:1.6.12 | | +--- com.google.guava:guava:20.0 -> 28.2-android | | | +--- com.google.guava:failureaccess:1.0.1 | | | +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava | | | +--- com.google.code.findbugs:jsr305:3.0.2 | | | +--- org.checkerframework:checker-compat-qual:2.5.5 | | | +--- com.google.errorprone:error_prone_annotations:2.3.4 | | | \--- com.google.j2objc:j2objc-annotations:1.3 | | +--- com.squareup.retrofit2:retrofit:2.5.0 | | | \--- com.squareup.okhttp3:okhttp:3.12.0 -> 3.12.2 | | | \--- com.squareup.okio:okio:1.15.0 | | +--- com.squareup.okhttp3:okhttp:3.12.2 (*) | | +--- com.squareup.okhttp3:logging-interceptor:3.12.2 -> 3.12.0 | | | \--- com.squareup.okhttp3:okhttp:3.12.0 -> 3.12.2 (*) | | +--- com.squareup.okhttp3:okhttp-urlconnection:3.12.2 -> 3.14.6 | | | \--- com.squareup.okhttp3:okhttp:3.14.6 -> 3.12.2 (*) | | +--- com.squareup.retrofit2:converter-jackson:2.5.0 | | | +--- com.squareup.retrofit2:retrofit:2.5.0 (*) | | | \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.10.2 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-joda:2.9.8 -> 2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | | \--- joda-time:joda-time:2.9.9 -> 2.10.5 | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | +--- io.reactivex:rxjava:1.3.8 | | +--- com.squareup.retrofit2:adapter-rxjava:2.4.0 | | | +--- com.squareup.retrofit2:retrofit:2.4.0 -> 2.5.0 (*) | | | \--- io.reactivex:rxjava:1.3.0 -> 1.3.8 | | +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.30 | | \--- com.microsoft.azure:azure-annotations:1.7.0 | +--- com.microsoft.azure:azure-cosmos:3.7.0 | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.1 -> 2.10.2 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1 -> 2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | +--- com.fasterxml.uuid:java-uuid-generator:3.2.0 | | +--- commons-io:commons-io:2.5 | | +--- commons-validator:commons-validator:1.6 | | | +--- commons-digester:commons-digester:1.8.1 | | | +--- commons-logging:commons-logging:1.2 | | | \--- commons-collections:commons-collections:3.2.2 | | +--- io.projectreactor:reactor-core:3.3.0.RELEASE -> 3.3.3.RELEASE (*) | | +--- io.netty:netty-codec-http:4.1.44.Final -> 4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final | | | | \--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-transport:4.1.45.Final | | | | +--- io.netty:netty-common:4.1.45.Final | | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | | \--- io.netty:netty-resolver:4.1.45.Final | | | | \--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-codec:4.1.45.Final | | | | +--- io.netty:netty-common:4.1.45.Final | | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | | \--- io.netty:netty-transport:4.1.45.Final (*) | | | \--- io.netty:netty-handler:4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | \--- io.netty:netty-codec:4.1.45.Final (*) | | +--- io.netty:netty-handler:4.1.44.Final -> 4.1.45.Final (*) | | +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30 | | +--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | +--- org.apache.commons:commons-collections4:4.2 | | +--- org.apache.commons:commons-text:1.6 | | | \--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | +--- com.google.guava:guava:25.0-jre -> 28.2-android (*) | | +--- io.dropwizard.metrics:metrics-core:4.1.0 -> 4.1.2 | | | \--- org.slf4j:slf4j-api:1.7.29 -> 1.7.30 | | +--- io.micrometer:micrometer-core:1.2.0 -> 1.3.2 | | | +--- org.hdrhistogram:HdrHistogram:2.1.11 | | | \--- org.latencyutils:LatencyUtils:2.0.3 | | +--- io.reactivex.rxjava2:rxjava:2.2.4 -> 2.2.17 (*) | | \--- io.projectreactor.netty:reactor-netty:0.9.4.RELEASE | | +--- io.netty:netty-codec-http:4.1.44.Final -> 4.1.45.Final (*) | | +--- io.netty:netty-codec-http2:4.1.44.Final -> 4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | +--- io.netty:netty-codec:4.1.45.Final (*) | | | +--- io.netty:netty-handler:4.1.45.Final (*) | | | \--- io.netty:netty-codec-http:4.1.45.Final (*) | | +--- io.netty:netty-handler:4.1.44.Final -> 4.1.45.Final (*) | | +--- io.netty:netty-handler-proxy:4.1.44.Final -> 4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | +--- io.netty:netty-codec:4.1.45.Final (*) | | | +--- io.netty:netty-codec-socks:4.1.45.Final | | | | +--- io.netty:netty-common:4.1.45.Final | | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | | \--- io.netty:netty-codec:4.1.45.Final (*) | | | \--- io.netty:netty-codec-http:4.1.45.Final (*) | | +--- io.netty:netty-transport-native-epoll:4.1.44.Final -> 4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | \--- io.netty:netty-transport-native-unix-common:4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | \--- io.netty:netty-transport:4.1.45.Final (*) | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | +--- com.microsoft.azure:applicationinsights-spring-boot-starter:2.5.1 | | +--- com.microsoft.azure:applicationinsights-core:2.5.1 | | \--- com.microsoft.azure:applicationinsights-web:2.5.1 | +--- com.microsoft.azure:azure-keyvault:1.2.2 | | +--- com.microsoft.azure:azure-keyvault-cryptography:1.2.2 | | | +--- com.microsoft.azure:azure-keyvault-webkey:1.2.2 | | | | +--- com.fasterxml.jackson.core:jackson-databind:2.9.9 -> 2.10.2 (*) | | | | +--- com.fasterxml.jackson.core:jackson-core:2.9.9 -> 2.10.2 | | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.9 -> 2.10.2 | | | | +--- commons-codec:commons-codec:1.10 -> 1.13 | | | | \--- com.google.guava:guava:24.1.1-jre -> 28.2-android (*) | | | +--- com.microsoft.azure:azure-keyvault-core:1.2.2 | | | | +--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | | | \--- com.google.guava:guava:24.1.1-jre -> 28.2-android (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.9.9 -> 2.10.2 (*) | | | +--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | | +--- commons-codec:commons-codec:1.10 -> 1.13 | | | \--- com.google.guava:guava:24.1.1-jre -> 28.2-android (*) | | +--- com.microsoft.azure:azure-keyvault-webkey:1.2.2 (*) | | +--- com.microsoft.azure:azure-client-runtime:1.6.12 (*) | | \--- com.microsoft.rest:client-runtime:1.6.12 (*) | +--- org.springframework.boot:spring-boot-starter-actuator:2.2.4.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-actuator-autoconfigure:2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot-actuator:2.2.4.RELEASE | | | | +--- org.springframework.boot:spring-boot:2.2.4.RELEASE (*) | | | | \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.2 (*) | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.4.RELEASE (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | +--- org.springframework:spring-context:5.2.3.RELEASE (*) | | | \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.2 (*) | | \--- io.micrometer:micrometer-core:1.3.2 (*) | +--- org.springframework.boot:spring-boot-starter-webflux:2.2.4.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-json:2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | | +--- org.springframework:spring-web:5.2.3.RELEASE (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.10.2 | | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.2 (*) | | | \--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | +--- org.springframework.boot:spring-boot-starter-reactor-netty:2.2.4.RELEASE | | | +--- io.projectreactor.netty:reactor-netty:0.9.4.RELEASE (*) | | | \--- org.glassfish:jakarta.el:3.0.3 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.4.RELEASE (*) | | +--- org.springframework:spring-web:5.2.3.RELEASE (*) | | +--- org.springframework:spring-webflux:5.2.3.RELEASE | | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | +--- org.springframework:spring-web:5.2.3.RELEASE (*) | | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | | \--- org.synchronoss.cloud:nio-multipart-parser:1.1.0 | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | \--- org.synchronoss.cloud:nio-stream-storage:1.1.3 | | \--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | +--- io.springfox:springfox-swagger2:3.0.0-SNAPSHOT | | +--- io.swagger:swagger-annotations:1.5.20 | | +--- io.swagger:swagger-models:1.5.20 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.5 -> 2.10.2 | | | +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.30 | | | \--- io.swagger:swagger-annotations:1.5.20 | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT | | | \--- io.springfox:springfox-core:3.0.0-SNAPSHOT | | | +--- net.bytebuddy:byte-buddy:1.9.10 -> 1.10.6 | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE | | | | +--- org.springframework:spring-beans:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-context:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-aop:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | \--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- io.springfox:springfox-schema:3.0.0-SNAPSHOT | | | +--- io.springfox:springfox-core:3.0.0-SNAPSHOT (*) | | | \--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | +--- io.springfox:springfox-swagger-common:3.0.0-SNAPSHOT | | | +--- io.swagger:swagger-annotations:1.5.20 | | | +--- io.swagger:swagger-models:1.5.20 (*) | | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | | +--- io.springfox:springfox-schema:3.0.0-SNAPSHOT (*) | | | +--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT | | | | +--- io.github.classgraph:classgraph:4.1.7 | | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | | | \--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | \--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | +--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT (*) | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | \--- org.mapstruct:mapstruct:1.3.1.Final | +--- io.springfox:springfox-spring-webflux:3.0.0-SNAPSHOT | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | \--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT (*) | +--- org.springframework.security:spring-security-oauth2-jose:5.3.0.RELEASE -> 5.2.1.RELEASE | | +--- com.nimbusds:nimbus-jose-jwt:7.8 -> 8.9 (*) | | +--- org.springframework.security:spring-security-core:5.2.1.RELEASE (*) | | +--- org.springframework.security:spring-security-oauth2-core:5.2.1.RELEASE | | | +--- org.springframework.security:spring-security-core:5.2.1.RELEASE (*) | | | +--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | \--- org.springframework:spring-web:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | \--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | +--- com.nimbusds:nimbus-jose-jwt:8.9 (*) | +--- com.onelogin:java-saml:2.5.0 | | +--- com.onelogin:java-saml-core:2.5.0 | | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | | +--- joda-time:joda-time:2.10.3 -> 2.10.5 | | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | | +--- org.apache.santuario:xmlsec:2.1.4 | | | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | | | | +--- com.fasterxml.woodstox:woodstox-core:5.0.3 | | | | | \--- org.codehaus.woodstox:stax2-api:3.1.4 | | | | \--- commons-codec:commons-codec:1.12 -> 1.13 | | | \--- commons-codec:commons-codec:1.12 -> 1.13 | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | +--- joda-time:joda-time:2.10.3 -> 2.10.5 | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | +--- org.apache.santuario:xmlsec:2.1.4 (*) | | \--- commons-codec:commons-codec:1.12 -> 1.13 | \--- io.reactivex:rxjava-reactive-streams:1.2.1 | +--- io.reactivex:rxjava:1.2.2 -> 1.3.8 | \--- org.reactivestreams:reactive-streams:1.0.0 -> 1.0.3 +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 (*) +--- org.jetbrains.kotlin:kotlin-reflect:1.3.70 (*) +--- org.springframework.boot:spring-boot-starter-actuator:2.2.4.RELEASE (*) +--- org.springframework.boot:spring-boot-starter-webflux:2.2.4.RELEASE (*) +--- org.springframework.cloud:spring-cloud-starter-config:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-starter:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE | | | \--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE -> 5.2.1.RELEASE | | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE | | | \--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE -> 5.2.1.RELEASE | | \--- org.springframework.security:spring-security-rsa:1.0.9.RELEASE | | \--- org.bouncycastle:bcpkix-jdk15on:1.64 | | \--- org.bouncycastle:bcprov-jdk15on:1.64 | +--- org.springframework.cloud:spring-cloud-config-client:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE (*) | | +--- org.springframework:spring-web:5.2.4.RELEASE -> 5.2.3.RELEASE (*) | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) +--- org.springframework.cloud:spring-cloud-config-server:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-config-client:2.2.2.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-actuator:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-web:2.2.5.RELEASE -> 2.2.4.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-json:2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-tomcat:2.2.4.RELEASE | | | +--- jakarta.annotation:jakarta.annotation-api:1.3.5 | | | +--- org.apache.tomcat.embed:tomcat-embed-core:9.0.30 | | | +--- org.apache.tomcat.embed:tomcat-embed-el:9.0.30 | | | \--- org.apache.tomcat.embed:tomcat-embed-websocket:9.0.30 | | | \--- org.apache.tomcat.embed:tomcat-embed-core:9.0.30 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.4.RELEASE (*) | | +--- org.springframework:spring-web:5.2.3.RELEASE (*) | | \--- org.springframework:spring-webmvc:5.2.3.RELEASE | | +--- org.springframework:spring-aop:5.2.3.RELEASE (*) | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | +--- org.springframework:spring-context:5.2.3.RELEASE (*) | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | +--- org.springframework:spring-expression:5.2.3.RELEASE (*) | | \--- org.springframework:spring-web:5.2.3.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-validation:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | +--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE -> 5.2.1.RELEASE | +--- org.springframework.security:spring-security-rsa:1.0.9.RELEASE (*) | +--- org.eclipse.jgit:org.eclipse.jgit:5.1.3.201810200350-r | | +--- com.jcraft:jsch:0.1.54 | | +--- com.jcraft:jzlib:1.1.1 | | +--- com.googlecode.javaewah:JavaEWAH:1.1.6 | | \--- org.slf4j:slf4j-api:1.7.2 -> 1.7.30 | +--- org.eclipse.jgit:org.eclipse.jgit.http.apache:5.1.3.201810200350-r | | +--- org.eclipse.jgit:org.eclipse.jgit:5.1.3.201810200350-r (*) | | +--- org.apache.httpcomponents:httpclient:4.5.5 -> 4.5.10 | | | +--- org.apache.httpcomponents:httpcore:4.4.12 -> 4.4.13 | | | +--- commons-logging:commons-logging:1.2 | | | \--- commons-codec:commons-codec:1.11 -> 1.13 | | \--- org.apache.httpcomponents:httpcore:4.4.9 -> 4.4.13 | +--- org.yaml:snakeyaml:1.25 | \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.2 | +--- org.yaml:snakeyaml:1.24 -> 1.25 | \--- com.fasterxml.jackson.core:jackson-core:2.10.2 +--- org.springframework.cloud:spring-cloud-starter-openfeign:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-starter:2.2.2.RELEASE (*) | +--- org.springframework.cloud:spring-cloud-openfeign-core:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-netflix-ribbon:2.2.2.RELEASE | | | \--- org.springframework.cloud:spring-cloud-netflix-archaius:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-starter-aop:2.2.5.RELEASE -> 2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | | +--- org.springframework:spring-aop:5.2.3.RELEASE (*) | | | \--- org.aspectj:aspectjweaver:1.9.5 | | \--- io.github.openfeign.form:feign-form-spring:3.8.0 | | +--- io.github.openfeign.form:feign-form:3.8.0 | | | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | | +--- org.springframework:spring-web:5.1.5.RELEASE -> 5.2.3.RELEASE (*) | | +--- commons-fileupload:commons-fileupload:1.4 | | | \--- commons-io:commons-io:2.2 -> 2.5 | | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | +--- org.springframework:spring-web:5.2.4.RELEASE -> 5.2.3.RELEASE (*) | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE (*) | +--- io.github.openfeign:feign-core:10.7.4 | +--- io.github.openfeign:feign-slf4j:10.7.4 | | +--- io.github.openfeign:feign-core:10.7.4 | | \--- org.slf4j:slf4j-api:1.7.13 -> 1.7.30 | \--- io.github.openfeign:feign-hystrix:10.7.4 | +--- io.github.openfeign:feign-core:10.7.4 | +--- com.netflix.archaius:archaius-core:0.7.6 | | +--- com.google.code.findbugs:jsr305:3.0.1 -> 3.0.2 | | +--- commons-configuration:commons-configuration:1.8 | | | +--- commons-lang:commons-lang:2.6 | | | \--- commons-logging:commons-logging:1.1.1 -> 1.2 | | +--- org.slf4j:slf4j-api:1.6.4 -> 1.7.30 | | +--- com.google.guava:guava:16.0 -> 28.2-android (*) | | +--- com.fasterxml.jackson.core:jackson-annotations:2.4.3 -> 2.10.2 | | +--- com.fasterxml.jackson.core:jackson-core:2.4.3 -> 2.10.2 | | \--- com.fasterxml.jackson.core:jackson-databind:2.4.3 -> 2.10.2 (*) | \--- com.netflix.hystrix:hystrix-core:1.5.18 | +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.30 | +--- com.netflix.archaius:archaius-core:0.4.1 -> 0.7.6 (*) | +--- io.reactivex:rxjava:1.2.0 -> 1.3.8 | \--- org.hdrhistogram:HdrHistogram:2.1.9 -> 2.1.11 +--- io.springfox:springfox-swagger2:3.0.0-SNAPSHOT (*) +--- io.springfox:springfox-spring-webflux:3.0.0-SNAPSHOT (*) +--- io.springfox:springfox-swagger-ui:3.0.0-SNAPSHOT +--- javax.persistence:persistence-api:1.0.2 +--- io.projectreactor:reactor-core:3.3.3.RELEASE (*) +--- com.microsoft.azure:applicationinsights-spring-boot-starter:2.5.1 (*) \--- com.squareup.okhttp3:okhttp:3.12.2 (*) ```

joshfree commented 4 years ago

Thanks for reporting this @allenhumphreys.

/cc @kushagraThapar @anuchandy @mikeharder

kushagraThapar commented 4 years ago

@allenhumphreys thanks for reporting this and putting out a great summary of the issue. We are working on this and plan to have a fix out there by the end of this week.

timwhit commented 4 years ago

Thanks, @kushagraThapar. Will there be a new release with this fix as well?

kushagraThapar commented 4 years ago

@timwhit yes, there will be.

kushagraThapar commented 4 years ago

This PR has fixed this issue: https://github.com/Azure/azure-sdk-for-java/pull/9211

New version will be released this week.

kushagraThapar commented 4 years ago

@allenhumphreys @timwhit - The fix for this has been released - v3.7.1 Please verify.

allenhumphreys commented 4 years ago

It will take some time to test this since we’ve switched over to gateway mode.

kushagraThapar commented 4 years ago

@allenhumphreys - Sure, please close this issue whenever you feel, or if you want, I can close it now, and you can re-open it if you see the issue again. However you want it.

kushagraThapar commented 4 years ago

Closing this, @allenhumphreys please reopen if you see the issue again.

allenhumphreys commented 4 years ago

@kushagraThapar I've updated my services and re-enabled DIRECT mode in my staging environment. I have observed what seems to be new behavior possibly related to this fix. I am receiving io.netty.util.IllegalReferenceCountException and the exception is being logged from RxGatewayStoreModel. Should I open a new ticket? I'm hesitant to enable this in production and I'd really like to because GATEWAY mode is not covered under the Cosmos SLA.

kushagraThapar commented 4 years ago

@allenhumphreys Yes, let's open a new ticket, so we can track it separately. Please paste the complete stack trace in addition to operation details and other important details that can help us. Meanwhile, continue using GATEWAY mode while we fix this.

timwhit commented 4 years ago

@kushagraThapar we were explicitly told yesterday by Azure Operations to NOT use Gateway mode as it has no SLA and there are no guarantees about it's uptime. This is an extremely serious issue and I'll be escalating the new ticket as soon as @allenhumphreys creates it.

kushagraThapar commented 4 years ago

@timwhit - as soon as @allenhumphreys creates the new ticket, we will start working on it. We are taking these fixes with high priority.

kushagraThapar commented 4 years ago

@timwhit - Regarding running on GATEWAY mode, I understand that it advised to NOT use GATEWAY mode, but the reason the mode still is in place is for situations like these. So it is perfectly okay to run on GATEWAY mode, until the DIRECT mode is completely fixed. Also, @allenhumphreys - Based on what you have mentioned so far (RxGatewayStoreModel) looks like you might be hitting the issue on HTTP layer, which might point to a memory leak issue in Http pipeline in reactor-netty. That bug is definitely not related to the recent fix we did, because the fix only touches Direct TCP protocol stack. That being said, I think you can even use DIRECT mode in your staging environment and keep and eye on memory and performance. Since in DIRECT mode (default is TCP) so HTTP's role is not a lot, which should not affect memory that much. Once you create the ticket, things will be more clear and I will be able to provide better recommendation.