Azure / azure-functions-dotnet-worker

Azure Functions out-of-process .NET language worker
MIT License
428 stars 182 forks source link

IFeatureCollection has been disposed exception during http trigger #2682

Open paulbatum opened 2 months ago

paulbatum commented 2 months ago

HTTP triggered .NET isolated function that uses an output binding to write to a queue will randomly fail with a 500, with the following exception:

Result: Failure Exception: System.ObjectDisposedException: IFeatureCollection has been disposed. Object name: 'Collection'.
    at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed()
    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.AspNetCoreHttpResponseHeadersCollection..ctor(HttpResponse request) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\HttpDataModel\AspNetCoreHttpHeadersCollection.cs:line 41
    at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
    at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
    at System.Lazy`1.CreateValue()
    at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToRpcHttpAsync(HttpResponseData response, ObjectSerializer serializer) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 104
    at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToRpcAsync(Object value, ObjectSerializer serializer) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 35
    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 102 Stack:
    at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed()
    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.AspNetCoreHttpResponseHeadersCollection..ctor(HttpResponse request) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\HttpDataModel\AspNetCoreHttpHeadersCollection.cs:line 41
    at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
    at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
    at System.Lazy`1.CreateValue()
    at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToRpcHttpAsync(HttpResponseData response, ObjectSerializer serializer) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 104
    at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToRpcAsync(Object value, ObjectSerializer serializer) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 35
    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 102

Investigative information

Multiple applications are hitting this error, but its a small number (less than 20 in last day).

All("FunctionsLogs")
| where PreciseTimeStamp >= datetime(2024-08-28T22:45:12.8953311Z) and PreciseTimeStamp <= datetime(2024-08-29T22:36:34.8464839Z) 
| where EventName == "FunctionCompleted"
| where ExceptionType == "Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException"
| where ExceptionMessage hasprefix "Result: Failure Exception: System.ObjectDisposedException: IFeatureCollection has been disposed. Object name: 'Collection'."
| summarize count() by EventPrimaryStampName, Role, AppName, HostVersion, Level, ExceptionMessage

Repro steps

Deploy net isolated app with following code and execute the http trigger multiple times.

Repros on both: 4.1035.2.2 4.34.2.2

        [Function("Function1_HttpStart")]
        public static async Task<MyOutput> HttpStart(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
            FunctionContext executionContext)
        {
            var body = req.Method switch
            {
                "GET" => $"Test message: {DateTime.UtcNow}",
                "POST" => await req.ReadAsStringAsync(),
                _ => throw new NotImplementedException()
            };

            var response = req.CreateResponse(System.Net.HttpStatusCode.OK);
            return new MyOutput
            {
                QueueOutput = body,
                HttpResponseData = response
            };
        }

        public class MyOutput
        {
            [QueueOutput("myqueue", Connection = "")]
            public string QueueOutput { get; set; }
            public HttpResponseData HttpResponseData { get; set; }

        }
paulbatum commented 2 months ago

@satvu figured out that this is related to the use of HttpRequestData in the context of an app that has a program.cs that initializes the ASP.NET integration e.g.

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

No repro once I switched the application code over to use HttpRequest:

        [Function("Function1_HttpStart")]
        public static async Task<MyOutput> HttpStart(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequest req,
            FunctionContext executionContext)
        {
            var body = req.Method switch
            {
                "GET" => $"Test message: {DateTime.UtcNow}",
                "POST" => await new StreamReader(req.Body).ReadToEndAsync(),
                _ => throw new NotImplementedException()
            };

            var response = new OkResult();
            return new MyOutput
            {
                QueueOutput = body,
                HttpResponse = response
            };
        }

        public class MyOutput
        {
            [QueueOutput("myqueue", Connection = "")]
            public string QueueOutput { get; set; }
            public ActionResult HttpResponse { get; set; }

        }
satvu commented 4 days ago

This is likely because in multi-output binding scenarios, this value is not nulled out like here when HttpResponseData is involved.