A health check request must be sent before each write operation. This will be implemented a lot like the RntbdContextRequest that we send when opening a channel.
Health check requests will be implemented together with a revised channel pool that's designed to "prime the request pipeline" with an initial health check. A channel will not be considered available until:
The connection is open
SSL negotiation is complete
Rntbd context request is complete
Initial "priming" health check is complete
Subsequently, a health check request will be issued before each write operation.
Test cases to be examined
TokenResolverTest::readDocumentThroughTokenResolver
Timeouts on initial point reads produce extra latency that should be addressed by this feature addition. We currently wait as long as 16 seconds. Goal: 2 seconds.
TokenResolveTest::deleteDocumentThroughTokenResolver
Timeouts on initial point deletes produce extra latency that should be addressed by this feature addition. We currently wait as long as 16 seconds. Goal: 2 seconds.
DocumentClientResourceLeakTest
Get to the bottom on two failure types: timeout and before/after memory deltas greater than the specified limit. See issue #188.
Issues to be addressed
We occasionally see a warning message about a memory leak in the SslHandler code following closure of an RntbdTransportClient. Long-running profiled benchmarks on Linux (on Azure) and macOS (locally) indicate that this memory is eventually released. Memory consumption does not appear to grow over time.
That said we should get to the bottom of this issue:
ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2125)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1327)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
A health check request must be sent before each write operation. This will be implemented a lot like the RntbdContextRequest that we send when opening a channel.
Health check requests will be implemented together with a revised channel pool that's designed to "prime the request pipeline" with an initial health check. A channel will not be considered available until:
Subsequently, a health check request will be issued before each write operation.
Test cases to be examined
TokenResolverTest::readDocumentThroughTokenResolver Timeouts on initial point reads produce extra latency that should be addressed by this feature addition. We currently wait as long as 16 seconds. Goal: 2 seconds.
TokenResolveTest::deleteDocumentThroughTokenResolver Timeouts on initial point deletes produce extra latency that should be addressed by this feature addition. We currently wait as long as 16 seconds. Goal: 2 seconds.
DocumentClientResourceLeakTest Get to the bottom on two failure types: timeout and before/after memory deltas greater than the specified limit. See issue #188.
Issues to be addressed
SslHandlercode following closure of anRntbdTransportClient. Long-running profiled benchmarks on Linux (on Azure) and macOS (locally) indicate that this memory is eventually released. Memory consumption does not appear to grow over time.That said we should get to the bottom of this issue: