search

Azure / azure-functions-servicebus-extension

Service Bus extension for Azure Functions
MIT License
65 stars 36 forks source link

System.ObjectDisposedException #78

Closed piotrzbyszynski closed 1 month ago

piotrzbyszynski commented 4 years ago

I have an Azure Function triggered by Service Bus Queue (consumption plan). Most of a time it works OK, but from time to time I get bunch of System.ObjectDisposedException exceptions.

I say bunch because in a very short period of time application insights logs few hundred of these. Here is detailed stack trace.

System.ObjectDisposedException:
   at Microsoft.Azure.ServiceBus.ClientEntity.ThrowIfClosed (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver+<ReceiveAsync>d__64.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver+<ReceiveAsync>d__62.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.ServiceBus.MessageReceivePump+<<MessagePumpTaskAsync>b__11_0>d.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)

Any help solving this issue would be highly appreciated

bm77525-kr commented 4 years ago

I'm seeing the same/similar thing on a premium plan running node functions:

System.ObjectDisposedException: Cannot access a disposed object.

Object name: 'Transaction'.

   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.DisposeMessagesAsync(IEnumerable`1 lockTokens, Outcome outcome)

   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)

   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)

   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.CompleteAsync(IEnumerable`1 lockTokens)

   at Microsoft.Azure.ServiceBus.MessageReceivePump.CompleteMessageIfNeededAsync(Message message)

The function is receiving messages (using the binding, not the SDK).

ravishankar54 commented 4 years ago

I'm also seeing the same/similar thing on a premium plan running node functions: System.ObjectDisposedException : Cannot access a disposed object. Object name: 'ServiceBusConnection has already been closed. Please create a new instance'. Failed method : Microsoft.Azure.ServiceBus.ServiceBusConnection.ThrowIfClosed

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d21.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 340) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 123) Inner exception System.ObjectDisposedException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw: at Microsoft.Azure.ServiceBus.ServiceBusConnection.ThrowIfClosed (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c) at Microsoft.Azure.ServiceBus.ClientEntity.ThrowIfClosed (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c) at Microsoft.Azure.ServiceBus.Core.MessageReceiver+d__68.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)

nforysinski commented 4 years ago

Can confirm this issue. Seeing these as long-idle functions first kick on with new queued requests as well as intermittently during normal function execution. Seems to not result in any actual function execution, simply abandons the message and another function or new host picks it up.


System.ObjectDisposedException:
   at Microsoft.Azure.ServiceBus.ServiceBusConnection.ThrowIfClosed (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at Microsoft.Azure.ServiceBus.ClientEntity.ThrowIfClosed (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver+<AbandonAsync>d__69.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.ServiceBus.MessageReceivePump+<AbandonMessageIfNeededAsync>d__15.MoveNext (Microsoft.Azure.ServiceBus, Version=4.1.1.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)```
jdimmerman commented 3 years ago

We're also seeing this. It happens rarely, but in the middle of function execution. In particular, I've seen it when we're attempting to complete a queue item via CompleteAsync(). Ultimately the queue item's lock expires, resulting in the message being retried and therefore processed twice.

Perhaps the function host is being shut down without properly draining? Any way to identify if that is happening?

jdimmerman commented 3 years ago

Is this a duplicate of https://github.com/Azure/azure-functions-host/issues/6027 and thus soon to be solved by https://github.com/Azure/azure-functions-servicebus-extension/issues/95?