MultiResponse with output queue leads to: Exception: System.ObjectDisposedException: IFeatureCollection has been disposed. Object name: 'Collection'

AnthonyGiretti commented 4 days ago

Description

I'm experiencing issues with my Azure Functions while using the MultiResponse on HttpTriggers. Once a single request it's working, once several requests are coming in, it leads to the following error: 'Exception while executing function: Functions.IqIngressFunction Result: Failure Exception: System.ObjectDisposedException: IFeatureCollection has been disposed. Object name: 'Collection'.'

Expected behavior

It should not throw that error

Actual behavior

It's working correctly locally on my machine, the issue appears only when it's deployed on Azure. When I remove ASP.NET Core Integration and returning only the messages (string[]) in output it's working When I remove the output queue it's also working (returning only IActionResult) Only when we mix Httpresponse management with ASP.NET Core Integration + output queue it's crashing on the cloud only, not locally

Relevant source code snippets

public class MultiResponse
{
    [QueueOutput("%SmsIngressQueueName%", Connection = Constants.StorageAccount.ConnectionString)]
    public string[] Messages { get; set; }

    [HttpResult]
    public IActionResult HttpResult { get; set; }
}

public class IqIngressFunction
{
    private readonly IIqSmsService _iqSmsService;
    private readonly ILogger<IqIngressFunction> _logger;

    public IqIngressFunction(IIqSmsService iqSmsService, ILogger<IqIngressFunction> logger)
    {
        _iqSmsService = iqSmsService;
        _logger = logger;
    }

    [Function(nameof(IqIngressFunction))]
    public async Task<MultiResponse> Run([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] HttpRequest req)
    {
        var output = new List<string>() { };
        var multiResponse = new MultiResponse();

        try
        {
            string requestBody = string.Empty;
            using (StreamReader streamReader = new StreamReader(req.Body))
            {
                requestBody = await streamReader.ReadToEndAsync();
            }

            _logger.LogInformation($"HTTP trigger function processed a request for IqIngressFunction, body: {requestBody}");

            var inbound = JsonConvert.DeserializeObject<IqInboundModel>(requestBody);
            if (inbound != null && inbound.deliveryReceipt)
            {
                multiResponse.Messages = new string[] { };
                multiResponse.HttpResult = new OkObjectResult("ok");
                return multiResponse;
            }
            var validMsg = _iqSmsService.ValidateIngressMessage(inbound);
            if (!string.IsNullOrEmpty(validMsg))
            {
                throw new Exception(validMsg);
            }

            var msg = new SmsMessage
            {
                Sender = inbound.from,
                Receiver = inbound.to[0],
                Message = inbound.text,
                Smscid = "IQ-x"
            };
            output.Add(JsonConvert.SerializeObject(msg));

            multiResponse.Messages = output.ToArray();
            multiResponse.HttpResult = new OkObjectResult("ok");
            return multiResponse;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, $"IqIngressFunction - An unexpected error occurred. Details: {ex.Message}");

            multiResponse.Messages = new string[] { };
            multiResponse.HttpResult = new BadRequestObjectResult(ex.Message);
            return multiResponse;
        }
    }
}

Known workarounds

Using a queue client (SDK required works)

App Details

<ItemGroup>
    <PackageReference Include="Azure.Identity" Version="1.12.0" />
  <PackageReference Include="Apache.NMS.ActiveMQ.NetStd" Version="1.8.0" />
  <PackageReference Include="Azure.Storage.Queues" Version="12.19.1" />
  <PackageReference Include="Microsoft.Azure.Cosmos" Version="3.42.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.3.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.5.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.4" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.23.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Warmup" Version="4.0.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.EventGrid" Version="3.4.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.EventHubs" Version="6.3.6" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.OpenApi" Version="1.5.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage" Version="6.6.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.4.0" />
    <PackageReference Include="Microsoft.Azure.AppConfiguration.Functions.Worker" Version="7.3.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.AzureAppConfiguration" Version="7.3.0" />
  <PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
  <PackageReference Include="Twilio" Version="5.79.0" />

.NET 8 Isolated mode
Azure function with HttpTrigger
Azure Functions runtime version v4:
*C#:

Screenshots

If deployed to Azure

see screenshot below

Timeframe issue observed::Happening right after the deployment happened, NOT HAPPENING ON LOCAL MACHIN

Steps to reproduce

Using MultiResponse as follows:

public class MultiResponse
{
    [QueueOutput("%SmsIngressQueueName%", Connection = Constants.StorageAccount.ConnectionString)]
    public string[] Messages { get; set; }

    [HttpResult]
    public IActionResult HttpResult { get; set; }
}

Then create an HttpClient and run 10 calls. 50 % of them will fail (InternlServerError 500), while others will work and return HttpStatus 200 OK

satvu commented 4 days ago

Adding some notes - this is similar to https://github.com/Azure/azure-functions-dotnet-worker/issues/2682, except it is not mitigated by using IActionResult. Stack trace points to serialization step for gRPC pipeline.

fabiocav commented 3 days ago

@satvu let's sync. This shouldn't be using the HttpResponseData path.

Azure / azure-functions-dotnet-worker