System.ObjectDisposedException: Cannot access a disposed object. on DefaultFunctionContext

[danielmarbach]

Description

When running the attached reproduction for a while (usually after a few hours) in functions using Linux (we tried it in Switzerland North) the following exception occurs

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.ServiceBusMessageHandler_workday_validation_queue
 ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
Exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope()
   at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a\_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 48
   at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack:    at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope()
   at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a\_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 48
   at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 101
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82
   at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:line 20
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 53
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 581
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 527
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 306
   --- End of inner exception stack trace ---
   at Microsoft.Azure.WebJobs.ServiceBus.SessionMessageProcessor.CompleteProcessingMessageAsync(ServiceBusSessionMessageActions actions, ServiceBusReceivedMessage message, FunctionResult result, CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.ProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
   at Azure.Messaging.ServiceBus.ServiceBusProcessor.OnProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
   at Azure.Messaging.ServiceBus.ServiceBusSessionProcessor.OnProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
   at Azure.Messaging.ServiceBus.SessionReceiverManager.OnMessageHandler(EventArgs args)
   at Azure.Messaging.ServiceBus.ReceiverManager.ProcessOneMessage(ServiceBusReceivedMessage triggerMessage, CancellationToken cancellationToken)

At the moment we don't think it is anything particular to do with the session in ASB or the ASB integration but rather with the function context handling in the middleware.

It could also be that it is related to the other issues in regard to ObjectDisposedExceptions.

https://github.com/Azure/azure-functions-dotnet-worker/issues/1929

I have already raised two PRs surrounding my research in this area but I'm unsure if they'll help

https://github.com/Azure/azure-functions-dotnet-worker/pull/2686 https://github.com/Azure/azure-functions-dotnet-worker/pull/2685

I have also made a comment about the use of TaskCompletionSource in the synchronization logic between the middleware.

Steps to reproduce

Run the repro on Service Bus for a while Experiment.zip

[jsparent]

We're getting the same kind of errors all over the place when using consumption plans; no behaviour so far that could help identify the faulting process. It fails about 5% of the time, randomly on our Azure Function App in consumption plan. Using App Service Plan on those function resolve the issue, but we would like to keep on using consumption plans.

Issue started a few weeks ago when updating references to latest versions.

Sample exception:

Result: Failure Exception: System.ObjectDisposedException: Cannot access a disposed object. Object name: 'IServiceProvider'. 
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException() 
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope() 
at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a\_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 48 
at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105 
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 91 
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88 Stack: 
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException() 
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope() 
at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a\_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 48 
at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105 
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 91 
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

[domenichelfenstein]

@danielmarbach 's issue was originally mine and he kinda posted it for me (thanks again)

I'm still getting the same exceptions you posted @jsparent However, I've tried publishing the function in Switzerland North and Europe West and I'm only getting the exceptions in Switzerland North, so far (running the function for two days now)

Where do you host your function, @jsparent ?

[jsparent]

@domenichelfenstein our function apps are located in Canada Central

[domenichelfenstein]

Are you using Windows or Linux, @jsparent ?

[jsparent]

@domenichelfenstein We're using Linux

[danr-stadion]

We are also experiencing the same issue. Our function app is running on a Linux-hosted consumption plan in the West Europe region.

For reference:

Result: Failure
Exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope()
at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a\_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 46
at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context)
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack: at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope()
at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a\_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 46
at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context)
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

[jbenettius]

Also experiencing the same error. Linux hosted consumption plan in East US. The function is queue triggered and has a queue output.

Exception: System.ObjectDisposedException: Cannot access a disposed object. Object name: 'IServiceProvider'. at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException() at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 48 at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c.b1_2(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 57 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89 Stack: at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException() at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 48 at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c.b1_2(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 57 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89

[jsparent]

UPDATE: We stopped having this error on September 21st 2024 from 18:00 EST. We didn't change anything, we even reverted back the original code afterwards, and we haven't had any problem since.

It happened on ~10 differents Function Apps, and it all stopped at once. While I'm glad this is no longer an issue, I'm pretty sure this is something happening in the "other" layer of the Function App Service. Hardware or routing maybe?

[domenichelfenstein]

Same here: Problem seems to be gone. But without an acknowledgement of the problem by Microsoft and a clear message what they've changed, I don't know if these exception could reappear all of a sudden.

[AlexMasson]

Same here: Problem seems to be gone. But without an acknowledgement of the problem by Microsoft and a clear message what they've changed, I don't know if these exception could reappear all of a sudden.

Unfortunately we encountered the error again a few hours ago (link issue).