Fix ByteBuf LEAK - Githubissues

carryxyh commented 5 years ago

最近使用arthas的时候，在arthas中发现了内存泄露，提高泄露级别后发现是在TelnetChannelHandler中导致的。

排查了一下，TelnetChannelHandler继承自ChannelInboundHandlerAdapter，这只是一个简单的适配器，不会自动释放buff。

Netty 默认使用池化-直接内存，这里release操作把这个buf归还到池子里比较好。

2019-05-14 11:31:52.545 ERROR [nioEventLoopGroup-2-4:i.n.u.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getBytes(AdvancedLeakAwareByteBuf.java:244)
        io.termd.core.telnet.netty.TelnetChannelHandler.channelRead(TelnetChannelHandler.java:45)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:563)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:745)
#2:
        Hint: 'TelnetChannelHandler#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:563)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:745)
#3:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:563)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:745)
#4:
        io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
        io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:563)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:745)
Created at:
        io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
        io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:563)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.lang.Thread.run(Thread.java:745)

CLAassistant commented 5 years ago

All committers have signed the CLA.

hengyunabc commented 5 years ago

怎样重现这个异常信息？之前遇到，没找到具体的原因： https://github.com/alibaba/arthas/issues/679

carryxyh commented 5 years ago

这个是偶现的。我也没有找到复现的方式。

我们一开始看到，以为是框架的问题，就把netty的泄露检测开到最高级。

后来发现我们的系统并没有问题，但是由于netty的泄露检测是全局变量，歪打正着的抓到了termd里这个类是最后的使用方。

alibaba / termd

Fix ByteBuf LEAK #16