search

Azure / DotNetty

DotNetty project – a port of netty, event-driven asynchronous network application framework
Other
4.09k stars 977 forks source link

memory leak issue #123

Closed caserzer closed 8 years ago

caserzer commented 8 years ago

I'm doing MQTT load test and the application report memory leak issue, I'm not sure whether it's my code issue or the MQTT decoder issue. C:\Debug\Debug>ProtocolGateway.Host.Console.exe ProviderId : d079e771-0495-4124-bd2f-ab63c2b50525 EventId : 5 Keywords : None Level : Error Message : Opcode : Info Task : 65529 Version : 0 Payload : [source : DotNetty.Common.ResourceLeakDetector] [message : LEAK: IByte Buffer.release() was not called before it's garbage-collected. See http://netty. io/wiki/reference-counted-objects.html for more information. Recent access records: 1

1:

at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at DotNetty.Common.ResourceLeakDetector.NewRecord(Object hint) at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak.RecordInternal(Ob ject hint) at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak.Record() at DotNetty.Buffers.AdvancedLeakAwareByteBuffer.WriteBytes(IByteBuffer src) at DotNetty.Codecs.ByteToMessageDecoder.ExpandCumulation(IByteBufferAllocator allocator, IByteBuffer cumulation, Int32 readable) at DotNetty.Codecs.ByteToMessageDecoder.<>c.<.cctor>b27_0(IByteBufferAlloca tor allocator, IByteBuffer cumulation, IByteBuffer input) at DotNetty.Codecs.ByteToMessageDecoder.ChannelRead(IChannelHandlerContext co ntext, Object message) at DotNetty.Transport.Channels.ChannelHandlerInvokerUtil.InvokeChannelReadNow (IChannelHandlerContext ctx, Object msg) at DotNetty.Transport.Channels.DefaultChannelHandlerInvoker.InvokeChannelRead (IChannelHandlerContext ctx, Object msg) at DotNetty.Transport.Channels.AbstractChannelHandlerContext.FireChannelRead( Object msg) at DotNetty.Transport.Channels.DefaultChannelPipeline.FireChannelRead(Object msg) at DotNetty.Transport.Channels.Sockets.AbstractSocketByteChannel.SocketByteCh annelUnsafe.FinishRead(SocketChannelAsyncOperation operation) at DotNetty.Transport.Channels.Sockets.AbstractSocketChannel.<>c.<.cctor>b__4 3_1(Object u, Object e) at DotNetty.Common.Concurrency.AbstractEventExecutor.StateActionWithContextTa skQueueNode.Run() at DotNetty.Common.Concurrency.SingleThreadEventExecutor.RunAllTasks(PreciseT imeSpan timeout) at DotNetty.Common.Concurrency.SingleThreadEventExecutor.b22_0() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute() at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCo ntext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution) at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task task) at DotNetty.Common.Concurrency.ExecutorTaskScheduler.QueueTask(Task task) at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate a ction, Object state, CancellationToken cancellationToken, TaskScheduler schedule r, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlM ark& stackMark) at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationTok en cancellationToken, TaskCreationOptions creationOptions, TaskScheduler schedul er) at DotNetty.Common.Concurrency.SingleThreadEventExecutor.Loop() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCo ntext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()Created at: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at DotNetty.Common.ResourceLeakDetector.NewRecord(Object hint) at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak..ctor(ResourceLea kDetector owner, Object referent) at DotNetty.Common.ResourceLeakDetector.Open(Object obj) at DotNetty.Buffers.AbstractByteBufferAllocator.ToLeakAwareBuffer(IByteBuffer buf) at DotNetty.Buffers.PooledByteBufferAllocator.NewBuffer(Int32 initialCapacity , Int32 maxCapacity) at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity)

at DotNetty.Codecs.ByteToMessageDecoder.ExpandCumulation(IByteBufferAllocator allocator, IByteBuffer cumulation, Int32 readable) at DotNetty.Codecs.ByteToMessageDecoder.<>c.<.cctor>b27_0(IByteBufferAlloca tor allocator, IByteBuffer cumulation, IByteBuffer input) at DotNetty.Codecs.ByteToMessageDecoder.ChannelRead(IChannelHandlerContext co ntext, Object message) at DotNetty.Transport.Channels.ChannelHandlerInvokerUtil.InvokeChannelReadNow (IChannelHandlerContext ctx, Object msg) at DotNetty.Transport.Channels.DefaultChannelHandlerInvoker.InvokeChannelRead (IChannelHandlerContext ctx, Object msg) at DotNetty.Transport.Channels.AbstractChannelHandlerContext.FireChannelRead( Object msg) at DotNetty.Transport.Channels.DefaultChannelPipeline.FireChannelRead(Object msg) at DotNetty.Transport.Channels.Sockets.AbstractSocketByteChannel.SocketByteCh annelUnsafe.FinishRead(SocketChannelAsyncOperation operation) at DotNetty.Transport.Channels.Sockets.AbstractSocketChannel.<>c.<.cctor>b__4 3_1(Object u, Object e) at DotNetty.Common.Concurrency.AbstractEventExecutor.StateActionWithContextTa skQueueNode.Run() at DotNetty.Common.Concurrency.SingleThreadEventExecutor.RunAllTasks(PreciseT imeSpan timeout) at DotNetty.Common.Concurrency.SingleThreadEventExecutor.b22_0() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute() at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCo ntext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution) at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task task) at DotNetty.Common.Concurrency.ExecutorTaskScheduler.QueueTask(Task task) at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate a ction, Object state, CancellationToken cancellationToken, TaskScheduler schedule r, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlM ark& stackMark) at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationTok en cancellationToken, TaskCreationOptions creationOptions, TaskScheduler schedul er) at DotNetty.Common.Concurrency.SingleThreadEventExecutor.Loop() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCo ntext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart] [exception : ] EventName : ErrorInfo Timestamp : 2016-06-04T18:02:26.4820537Z ProcessId : 4212 ThreadId : 1820

ProviderId : d079e771-0495-4124-bd2f-ab63c2b50525 EventId : 5 Keywords : None Level : Error Message : Opcode : Info Task : 65529 Version : 0 Payload : [source : DotNetty.Common.ResourceLeakDetector] [message : LEAK: IByte Buffer.release() was not called before it's garbage-collected. See http://netty. io/wiki/reference-counted-objects.html for more information. Recent access records: 1

1:

    Hint: 'MqttDecoder#0' will handle the message from this point.

at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at DotNetty.Common.ResourceLeakDetector.NewRecord(Object hint) at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak.RecordInternal(Ob ject hint) at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak.Record(Object hin t) at DotNetty.Buffers.AdvancedLeakAwareByteBuffer.Touch(Object hint) at DotNetty.Common.Utilities.ReferenceCountUtil.Touch[T](T msg, Object hint) at DotNetty.Transport.Channels.AbstractChannelHandlerContext.FireChannelRead( Object msg) at DotNetty.Transport.Channels.DefaultChannelPipeline.FireChannelRead(Object msg) at DotNetty.Transport.Channels.Sockets.AbstractSocketByteChannel.SocketByteCh annelUnsafe.FinishRead(SocketChannelAsyncOperation operation) at DotNetty.Transport.Channels.Sockets.AbstractSocketChannel.<>c.<.cctor>b__4 3_1(Object u, Object e) at DotNetty.Common.Concurrency.AbstractEventExecutor.StateActionWithContextTa skQueueNode.Run() at DotNetty.Common.Concurrency.SingleThreadEventExecutor.RunAllTasks(PreciseT imeSpan timeout) at DotNetty.Common.Concurrency.SingleThreadEventExecutor.b__22_0() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute() at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCo ntext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution) at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task task) at DotNetty.Common.Concurrency.ExecutorTaskScheduler.QueueTask(Task task) at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate a ction, Object state, CancellationToken cancellationToken, TaskScheduler schedule r, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlM ark& stackMark) at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationTok en cancellationToken, TaskCreationOptions creationOptions, TaskScheduler schedul er) at DotNetty.Common.Concurrency.SingleThreadEventExecutor.Loop() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCo ntext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()Created at: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at DotNetty.Common.ResourceLeakDetector.NewRecord(Object hint) at DotNetty.Common.ResourceLeakDetector.DefaultResourceLeak..ctor(ResourceLea kDetector owner, Object referent) at DotNetty.Common.ResourceLeakDetector.Open(Object obj) at DotNetty.Buffers.AbstractByteBufferAllocator.ToLeakAwareBuffer(IByteBuffer buf) at DotNetty.Buffers.PooledByteBufferAllocator.NewBuffer(Int32 initialCapacity , Int32 maxCapacity) at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity)

at DotNetty.Transport.Channels.FixedRecvByteBufAllocator.HandleImpl.Allocate( IByteBufferAllocator alloc) at DotNetty.Transport.Channels.Sockets.AbstractSocketByteChannel.SocketByteCh annelUnsafe.FinishRead(SocketChannelAsyncOperation operation) at DotNetty.Transport.Channels.Sockets.AbstractSocketChannel.<>c.<.cctor>b__4 3_1(Object u, Object e) at DotNetty.Common.Concurrency.AbstractEventExecutor.StateActionWithContextTa skQueueNode.Run() at DotNetty.Common.Concurrency.SingleThreadEventExecutor.RunAllTasks(PreciseT imeSpan timeout) at DotNetty.Common.Concurrency.SingleThreadEventExecutor.b__22_0() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute() at System.Threading.Tasks.Task.ExecutionContextCallback(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCo ntext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution) at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task task) at DotNetty.Common.Concurrency.ExecutorTaskScheduler.QueueTask(Task task) at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate a ction, Object state, CancellationToken cancellationToken, TaskScheduler schedule r, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlM ark& stackMark) at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationTok en cancellationToken, TaskCreationOptions creationOptions, TaskScheduler schedul er) at DotNetty.Common.Concurrency.SingleThreadEventExecutor.Loop() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCo ntext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, C ontextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart] [exception : ] EventName : ErrorInfo Timestamp : 2016-06-04T18:02:26.5914240Z ProcessId : 4212 ThreadId : 1820

nayato commented 8 years ago

@caserzer, a few questions:

  1. what version of DotNetty are you using?
  2. what's in the pipeline? TLS? MQTT decoder?
  3. are you using Protocol Gateway without any modifications?
caserzer commented 8 years ago

@nayato

  1. DotNetty 0.3.0
  2. No TLS , only MQTT decoder
  3. Using TransientSessionStatePersistenceProvider, no qos2 provider , no authentication , remove send message to IOT Hub , only enqueue
  4. Connections count 1: OK Connections count 10 : OK Connections count 50 : OK Connections count 100 : report memory leak MQTT message 1K , every connection will send 100 messages including connect. the leak only occurred when sending connect command
nayato commented 8 years ago

can you post a repro? That would help speed up the analysis

nayato commented 8 years ago

also, when you say "only enqueue", do you release the MQTT packets / payloads? normally it is done in protocol gateway here: https://github.com/Azure/azure-iot-protocol-gateway/blob/master/src/ProtocolGateway.Core/Mqtt/MqttIotHubAdapter.cs#L335

caserzer commented 8 years ago

@nayato thanks for helping. I found in my modification I simply comment theusing (Stream bodyStream = packet.Payload.IsReadable() ? new ReadOnlyByteBufferStream(packet.Payload, true) : null) which cause the memory leak. Thanks