search

Azure / DotNetty

DotNetty project – a port of netty, event-driven asynchronous network application framework
Other
4.09k stars 977 forks source link

Writing message and closing channel in ChannelActive causes leak #325

Closed Rumpel78 closed 6 years ago

Rumpel78 commented 6 years ago

I have following pipeline:

var STRING_ENCODER = new StringEncoder(Encoding.UTF8);
var STRING_DECODER = new StringDecoder(Encoding.UTF8);
ResourceLeakDetector.Level = ResourceLeakDetector.DetectionLevel.Paranoid;

var bootstrap = new ServerBootstrap();
bootstrap
    .Group(_bossGroup, _workerGroup)
    .ChannelFactory(() => new TcpServerSocketChannel(AddressFamily.InterNetwork))
    .Handler(new LoggingHandler("Channel", LogLevel.DEBUG))
    .ChildHandler(new ActionChannelInitializer<ISocketChannel>(channel =>
    {
        var pipeline = channel.Pipeline;
        pipeline.AddLast(new LineBasedFrameDecoder(MAX_MESSAGE_SIZE, true, true));
        pipeline.AddLast(STRING_ENCODER, STRING_DECODER);
        pipeline.AddLast(new DummyHandler());
    }));

DummyHandler:

public class DummyHandler : SimpleChannelInboundHandler<string>
{
   private static readonly ILog Log = LogManager.GetLogger(typeof(DummyHandler));

   protected override void ChannelRead0(IChannelHandlerContext ctx, string msg)
   {
      Log.Debug(msg);
   }

   public override void ChannelActive(IChannelHandlerContext context)
   {
      context.WriteAndFlushAsync("Bye");
      context.CloseAsync();
   }
}

I always get following message after ~2000-3000 connections:


2017-12-12 12:14:38,726 [2] ERROR DotNettyLogger.DotNetty.Common.ResourceLeakDetector - DotNetty.Common.ResourceLeakDetector - LEAK: IByteBuffer.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 1
#1:
    Hint: 'HeadContext#0' will handle the message from this point.
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at DotNetty.Common.ResourceLeakDetector.NewRecord(Object hint)
   at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak.RecordInternal(Object hint)
   at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak.Record(Object hint)
   at DotNetty.Buffers.AdvancedLeakAwareByteBuffer.Touch(Object hint)
   at DotNetty.Common.Utilities.ReferenceCountUtil.Touch[T](T msg, Object hint)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.WriteAsync(Object msg, Boolean flush)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.WriteAsync(Object msg)
   at DotNetty.Codecs.MessageToMessageEncoder`1.WriteAsync(IChannelHandlerContext ctx, Object msg)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeWriteAsync0(Object msg)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeWriteAndFlushAsync(Object msg)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.WriteAsync(Object msg, Boolean flush)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.WriteAndFlushAsync(Object message)
   at MyProject.ChannelHandler.DummyHandler.ChannelActive(IChannelHandlerContext context) in C:\Users\Administrator\Documents\Visual Studio 2017\Projects\MyProject\ChannelHandler\DummyHandler.cs:line 24
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive()
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive(AbstractChannelHandlerContext next)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.FireChannelActive()
   at DotNetty.Transport.Channels.ChannelHandlerAdapter.ChannelActive(IChannelHandlerContext context)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive()
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive(AbstractChannelHandlerContext next)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.FireChannelActive()
   at DotNetty.Transport.Channels.ChannelHandlerAdapter.ChannelActive(IChannelHandlerContext context)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive()
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive(AbstractChannelHandlerContext next)
   at DotNetty.Transport.Channels.DefaultChannelPipeline.FireChannelActive()
   at DotNetty.Transport.Channels.AbstractChannel.AbstractUnsafe.Register0(TaskCompletionSource promise)
   at DotNetty.Transport.Channels.AbstractChannel.AbstractUnsafe.<>c.<RegisterAsync>b__11_0(Object u, Object p)
   at DotNetty.Common.Concurrency.AbstractEventExecutor.StateActionWithContextTaskQueueNode.Run()
   at DotNetty.Common.Concurrency.SingleThreadEventExecutor.RunAllTasks(PreciseTimeSpan timeout)
   at DotNetty.Common.Concurrency.SingleThreadEventExecutor.<Loop>b__25_0()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task task)
   at DotNetty.Common.Concurrency.ExecutorTaskScheduler.QueueTask(Task task)
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
   at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationToken cancellationToken, TaskCreationOptions creationOptions, TaskScheduler scheduler)
   at DotNetty.Common.Concurrency.SingleThreadEventExecutor.Loop()
   at DotNetty.Common.Concurrency.XThread.<>c__DisplayClass10_0.<.ctor>b__0(Object x)
   at DotNetty.Common.Concurrency.XThread.<>c__DisplayClass13_0.<CreateLongRunningTask>b__0()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork(Object obj)
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)
Created at:
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at DotNetty.Common.ResourceLeakDetector.NewRecord(Object hint)
   at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak..ctor(ResourceLeakDetector owner, Object referent)
   at DotNetty.Common.ResourceLeakDetector.Track(Object obj)
   at DotNetty.Buffers.AbstractByteBufferAllocator.ToLeakAwareBuffer(IByteBuffer buf)
   at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
   at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity)
   at DotNetty.Buffers.ByteBufferUtil.EncodeString0(IByteBufferAllocator alloc, Boolean enforceHeap, String src, Encoding encoding, Int32 extraCapacity)
   at DotNetty.Codecs.StringEncoder.Encode(IChannelHandlerContext context, String message, List`1 output)
   at DotNetty.Codecs.MessageToMessageEncoder`1.WriteAsync(IChannelHandlerContext ctx, Object msg)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeWriteAsync0(Object msg)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeWriteAndFlushAsync(Object msg)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.WriteAsync(Object msg, Boolean flush)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.WriteAndFlushAsync(Object message)
   at MyProject.ChannelHandler.DummyHandler.ChannelActive(IChannelHandlerContext context) in C:\Users\Administrator\Documents\Visual Studio 2017\Projects\MyProject\ChannelHandler\DummyHandler.cs:line 24
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive()
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive(AbstractChannelHandlerContext next)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.FireChannelActive()
   at DotNetty.Transport.Channels.ChannelHandlerAdapter.ChannelActive(IChannelHandlerContext context)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive()
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive(AbstractChannelHandlerContext next)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.FireChannelActive()
   at DotNetty.Transport.Channels.ChannelHandlerAdapter.ChannelActive(IChannelHandlerContext context)
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive()
   at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelActive(AbstractChannelHandlerContext next)
   at DotNetty.Transport.Channels.DefaultChannelPipeline.FireChannelActive()
   at DotNetty.Transport.Channels.AbstractChannel.AbstractUnsafe.Register0(TaskCompletionSource promise)
   at DotNetty.Transport.Channels.AbstractChannel.AbstractUnsafe.<>c.<RegisterAsync>b__11_0(Object u, Object p)
   at DotNetty.Common.Concurrency.AbstractEventExecutor.StateActionWithContextTaskQueueNode.Run()
   at DotNetty.Common.Concurrency.SingleThreadEventExecutor.RunAllTasks(PreciseTimeSpan timeout)
   at DotNetty.Common.Concurrency.SingleThreadEventExecutor.<Loop>b__25_0()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task task)
   at DotNetty.Common.Concurrency.ExecutorTaskScheduler.QueueTask(Task task)
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
   at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationToken cancellationToken, TaskCreationOptions creationOptions, TaskScheduler scheduler)
   at DotNetty.Common.Concurrency.SingleThreadEventExecutor.Loop()
   at DotNetty.Common.Concurrency.XThread.<>c__DisplayClass10_0.<.ctor>b__0(Object x)
   at DotNetty.Common.Concurrency.XThread.<>c__DisplayClass13_0.<CreateLongRunningTask>b__0()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork(Object obj)
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)

If I remove the WriteAndFlushAsync call, this message does not appear. I suspected the channel being closed before the ouput could be written, so I have tried calling WriteAndFlushAsync with await, Task.WaitAll and WriteAndFlushAsync("").Wait() but the result is always the same.

I am trying to implement an IP-Block - a handler testing the connected IP and closing the channel with an error message. I would like to avoid a memory leak.

Any ideas on this?

StormHub commented 6 years ago

What's the version you have this issue?

Rumpel78 commented 6 years ago

I am using latest NuGet Version in production and tried the latest github version for testing.

Essentially I have stripped away all code, tried to pinpoint my error and now I think this happens somewhere in the StringEncoder/Decoder, or, perhaps, this is not really an error.

StormHub commented 6 years ago

@Rumpel78 I think I found the root cause of the false positive leak report.

Rumpel78 commented 6 years ago

@StormHub Looks good, no more messages - thanks!