search

grpc / grpc-java

The Java gRPC implementation. HTTP/2 based RPC
https://grpc.io/docs/languages/java/
Apache License 2.0
11.26k stars 3.79k forks source link

Resource leak in netty tests #3353

Open carl-mastrangelo opened 6 years ago

carl-mastrangelo commented 6 years ago

I haven't tracked down the responsible, but I ran with leak detection and found this:

Aug 16, 2017 10:58:18 AM io.netty.util.ResourceLeakDetector reportTracedLeak
SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
WARNING: 3 leak records were discarded because the leak record count is limited to 4. Use system property io.netty.leakDetection.maxRecords to increase the limit.
Recent access records: 4
#4:
    io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:936)
    io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
    io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:109)
    io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:256)
    io.netty.channel.embedded.EmbeddedChannel.doWrite(EmbeddedChannel.java:723)
    io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:872)
    io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:839)
    io.netty.channel.embedded.EmbeddedChannel$EmbeddedUnsafe$1.flush(EmbeddedChannel.java:819)
    io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1321)
    io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
    io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
    io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
    io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:191)
    io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:509)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:373)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1364)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380)
    io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:941)
    io.netty.channel.embedded.EmbeddedChannel.flushInbound(EmbeddedChannel.java:359)
    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:320)
    io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
    io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
    io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
    org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
    org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
    org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
    org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    org.junit.runners.ParentRunner$3.run(ParentRunner.java:240)
    org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:65)
    org.junit.runners.ParentRunner.runChildren(ParentRunner.java:238)
    org.junit.runners.ParentRunner.access$000(ParentRunner.java:55)
    org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:231)
    org.junit.runners.ParentRunner.run(ParentRunner.java:316)
#3:
    io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:924)
    io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:35)
    io.netty.util.ReferenceCountUtil.retain(ReferenceCountUtil.java:36)
    io.netty.channel.embedded.EmbeddedChannel.doWrite(EmbeddedChannel.java:721)
    io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:872)
    io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:839)
    io.netty.channel.embedded.EmbeddedChannel$EmbeddedUnsafe$1.flush(EmbeddedChannel.java:819)
    io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1321)
    io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
    io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
    io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
    io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:191)
    io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:509)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:373)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1364)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380)
    io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:941)
    io.netty.channel.embedded.EmbeddedChannel.flushInbound(EmbeddedChannel.java:359)
    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:320)
    io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
    io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
    io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
    org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
    org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
#2:
    Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:810)
    io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeSettingsAck(DefaultHttp2FrameWriter.java:260)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.CGLIB$writeSettingsAck$6(<generated>)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d$$FastClassByMockitoWithCGLIB$$af30f311.invoke(<generated>)
    org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216)
    org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10)
    org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22)
    org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27)
    org.mockito.internal.invocation.InvocationImpl.callRealMethod(InvocationImpl.java:108)
    org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36)
    org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:93)
    org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
    org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
    org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.writeSettingsAck(<generated>)
    io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeSettingsAck(DecoratingHttp2FrameWriter.java:75)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeSettingsAck(DefaultHttp2ConnectionEncoder.java:264)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:419)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:635)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.readSettingsFrame(DefaultHttp2FrameReader.java:542)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:263)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
    io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:373)
    io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:244)
    io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:433)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:492)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:431)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:317)
    io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
    io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
    io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
    org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
    org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
    org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)

#1:
    io.netty.buffer.AdvancedLeakAwareByteBuf.writeInt(AdvancedLeakAwareByteBuf.java:558)
    io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:260)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeSettingsAck(DefaultHttp2FrameWriter.java:259)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.CGLIB$writeSettingsAck$6(<generated>)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d$$FastClassByMockitoWithCGLIB$$af30f311.invoke(<generated>)
    org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216)
    org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10)
    org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22)
    org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27)
    org.mockito.internal.invocation.InvocationImpl.callRealMethod(InvocationImpl.java:108)
    org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36)
    org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:93)
    org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
    org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
    org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.writeSettingsAck(<generated>)
    io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeSettingsAck(DecoratingHttp2FrameWriter.java:75)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeSettingsAck(DefaultHttp2ConnectionEncoder.java:264)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:419)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:635)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.readSettingsFrame(DefaultHttp2FrameReader.java:542)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:263)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
    io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:373)
    io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:244)
    io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:433)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:492)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:431)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:317)
    io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
    io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
    io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
    org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
    org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
    org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
    org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

Created at:
    io.netty.util.ResourceLeakDetector.track(ResourceLeakDetector.java:237)
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:172)
    io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:109)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeSettingsAck(DefaultHttp2FrameWriter.java:258)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.CGLIB$writeSettingsAck$6(<generated>)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d$$FastClassByMockitoWithCGLIB$$af30f311.invoke(<generated>)
    org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216)
    org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10)
    org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22)
    org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27)
    org.mockito.internal.invocation.InvocationImpl.callRealMethod(InvocationImpl.java:108)
    org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36)
    org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:93)
    org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
    org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
    org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
    io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.writeSettingsAck(<generated>)
    io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeSettingsAck(DecoratingHttp2FrameWriter.java:75)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeSettingsAck(DefaultHttp2ConnectionEncoder.java:264)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:419)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:635)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.readSettingsFrame(DefaultHttp2FrameReader.java:542)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:263)
    io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
    io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
    io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:373)
    io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:244)
    io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:433)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:492)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:431)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:317)
    io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
    io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
    io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
    org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
    org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
    org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
    org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
alexvas commented 6 years ago

BTW, I see slightly different stacktrace at our production grpc-based backend:

[07:12:15,418] [grpc-default-worker-ELG-3-11] ERROR io.netty.util.ResourceLeakDetector ~ LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://n
etty.io/wiki/reference-counted-objects.html for more information.
WARNING: 107 leak records were discarded because the leak record count is limited to 4. Use system property io.netty.leakDetection.maxRecords to increase the limit.
Recent access records: 4
#4:
        io.netty.buffer.AdvancedLeakAwareByteBuf.readBytes(AdvancedLeakAwareByteBuf.java:498)
        io.grpc.netty.NettyReadableBuffer.readBytes(NettyReadableBuffer.java:60)
        io.grpc.internal.CompositeReadableBuffer$3.readInternal(CompositeReadableBuffer.java:95)
        io.grpc.internal.CompositeReadableBuffer$ReadOperation.read(CompositeReadableBuffer.java:216)
        io.grpc.internal.CompositeReadableBuffer.execute(CompositeReadableBuffer.java:174)
        io.grpc.internal.CompositeReadableBuffer.readBytes(CompositeReadableBuffer.java:91)
        io.grpc.internal.ReadableBuffers$BufferInputStream.read(ReadableBuffers.java:327)
        io.grpc.protobuf.lite.ProtoLiteUtils$2.parse(ProtoLiteUtils.java:133)
        io.grpc.protobuf.lite.ProtoLiteUtils$2.parse(ProtoLiteUtils.java:79)
        io.grpc.MethodDescriptor.parseRequest(MethodDescriptor.java:280)
        io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messageRead(ServerCallImpl.java:246)
        io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1.runInContext(ServerImpl.java:571)
        io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        java.lang.Thread.run(Thread.java:748)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:402)
        io.grpc.netty.NettyReadableBuffer.readUnsignedByte(NettyReadableBuffer.java:55)
        io.grpc.internal.CompositeReadableBuffer$1.readInternal(CompositeReadableBuffer.java:71)
        io.grpc.internal.CompositeReadableBuffer$ReadOperation.read(CompositeReadableBuffer.java:216)
        io.grpc.internal.CompositeReadableBuffer.execute(CompositeReadableBuffer.java:174)
        io.grpc.internal.CompositeReadableBuffer.readUnsignedByte(CompositeReadableBuffer.java:74)
        io.grpc.internal.AbstractReadableBuffer.readInt(AbstractReadableBuffer.java:48)
        io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:330)
        io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:232)
        io.grpc.internal.MessageDeframer.request(MessageDeframer.java:143)
        io.grpc.internal.AbstractStream$TransportState.requestMessagesFromDeframer(AbstractStream.java:195)
        io.grpc.netty.NettyServerStream$Sink$1.run(NettyServerStream.java:90)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        java.lang.Thread.run(Thread.java:748)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:402)
        io.grpc.netty.NettyReadableBuffer.readUnsignedByte(NettyReadableBuffer.java:55)
        io.grpc.internal.CompositeReadableBuffer$1.readInternal(CompositeReadableBuffer.java:71)
        io.grpc.internal.CompositeReadableBuffer$ReadOperation.read(CompositeReadableBuffer.java:216)
        io.grpc.internal.CompositeReadableBuffer.execute(CompositeReadableBuffer.java:174)
        io.grpc.internal.CompositeReadableBuffer.readUnsignedByte(CompositeReadableBuffer.java:74)
        io.grpc.internal.AbstractReadableBuffer.readInt(AbstractReadableBuffer.java:47)
        io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:330)
        io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:232)
        io.grpc.internal.MessageDeframer.request(MessageDeframer.java:143)
        io.grpc.internal.AbstractStream$TransportState.requestMessagesFromDeframer(AbstractStream.java:195)
        io.grpc.netty.NettyServerStream$Sink$1.run(NettyServerStream.java:90)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        java.lang.Thread.run(Thread.java:748)
#1:
        io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:402)
        io.grpc.netty.NettyReadableBuffer.readUnsignedByte(NettyReadableBuffer.java:55)
        io.grpc.internal.CompositeReadableBuffer$1.readInternal(CompositeReadableBuffer.java:71)
        io.grpc.internal.CompositeReadableBuffer$ReadOperation.read(CompositeReadableBuffer.java:216)
        io.grpc.internal.CompositeReadableBuffer.execute(CompositeReadableBuffer.java:174)
        io.grpc.internal.CompositeReadableBuffer.readUnsignedByte(CompositeReadableBuffer.java:74)
        io.grpc.internal.AbstractReadableBuffer.readInt(AbstractReadableBuffer.java:46)
        io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:330)
        io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:232)
        io.grpc.internal.MessageDeframer.request(MessageDeframer.java:143)
        io.grpc.internal.AbstractStream$TransportState.requestMessagesFromDeframer(AbstractStream.java:195)
        io.grpc.netty.NettyServerStream$Sink$1.run(NettyServerStream.java:90)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        java.lang.Thread.run(Thread.java:748)
Created at:
        io.netty.util.ResourceLeakDetector.track(ResourceLeakDetector.java:237)
        io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:172)
        io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:80)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:122)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        java.lang.Thread.run(Thread.java:748)
carl-mastrangelo commented 6 years ago

@alexvas What you are seeing is a different bug. Would you mind filing a separate issue? This one is specifically for tests. (and thus not as urgent as yours)

@ericgribkoff Message deframing change possibly?

alexvas commented 6 years ago

@carl-mastrangelo done in #3422

vinodhabib commented 3 days ago

@kannanjgithub As per the defect details I can run the whole test class as well as single unit test (NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail()) which is mentioned in error logs of the defect successfully.

PFA the Successful Test class which ran in my local machine. image

Even I tried some of the approaches/details which is mentioned in the below documentation for finding the Resource leaks in unit tests but unable to see any. https://netty.io/wiki/reference-counted-objects.html

Also has you mentioned earlier some classes are mocked which are going to used in the setup() method the get required streamListner/streamTraces object instance from Factory which is required by all the tests otherwise we will be getting NPE saying streamTracerFactory is null.

Need to have a connect and have more discussion on the same with you to make sure I am not missing anything to see this issue

vinodhabib commented 2 days ago

As per the suggestion from @kannanjgithub working in the direction to enable to info/debug logs through logback.xml for the UT case to re-produce issue and the same in progress.