search

Azure / Azure-Functions

1.11k stars 196 forks source link

The function invocation context is missing the forwarding task property #2481

Closed dzejsien closed 3 months ago

dzejsien commented 4 months ago

"FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",

<TargetFramework>net8.0</TargetFramework>
<AzureFunctionsVersion>v4</AzureFunctionsVersion>
Packages:

    <PackageReference Include="Azure.Identity" Version="1.11.3" />
    <PackageReference Include="Azure.Messaging.EventHubs" Version="5.11.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.EventHubs" Version="6.3.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />

ISSUE:

I have 2 functions:

Startup:

var host = new HostBuilder()
    .ConfigureAppConfiguration((_, config) =>
        config
            .AddJsonFile("appsettings.json", true, true)
            .AddDefaultAzureAppConfiguration(...)
            .AddUserSecrets(Assembly.GetExecutingAssembly(), true))
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
        services.Configure<LoggerFilterOptions>(options =>
        {
            ...
        });

        //Add healthchecks
        services
        .AddKeyedSingleton(serviceKey: EventHubServiceNames.DeadletterEventHub, (serviceProvider, serviceKey) =>
        {
            var configuration = serviceProvider.GetRequiredService<IConfiguration>();
            return new EventHubProducerClient(configuration["..."]);
        })
        .AddKeyedSingleton(serviceKey: EventHubServiceNames.OutputEventHub, (serviceProvider, serviceKey) =>
        {
            var configuration = serviceProvider.GetRequiredService<IConfiguration>();
            return new EventHubProducerClient(configuration["..."], configuration["..."]);
        })
        .AddKeyedSingleton(serviceKey: EventHubServiceNames.InputEventHub, (serviceProvider, serviceKey) =>
        {
            var configuration = serviceProvider.GetRequiredService<IConfiguration>();
            return new EventHubProducerClient(configuration["..."], configuration["..."]);
        })
        .AddHealthChecks()
        .AddAzureEventHub(clientFactory: serviceProvider => serviceProvider.GetRequiredKeyedService<EventHubProducerClient>(EventHubServiceNames.InputEventHub), name: "inEventHub")
        .AddAzureEventHub(clientFactory: serviceProvider => serviceProvider.GetRequiredKeyedService<EventHubProducerClient>(EventHubServiceNames.OutputEventHub), name: "outEventHub")
        .AddAzureEventHub(clientFactory: serviceProvider => serviceProvider.GetRequiredKeyedService<EventHubProducerClient>(EventHubServiceNames.DeadletterEventHub), name: "dlqEventHub");

    })
    .ConfigureLogging((hostingContext, logging) =>
    {
        logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
    })
    .Build();

var lifetime = host.Services.GetRequiredService<IHostApplicationLifetime>();
var loggerFactory = host.Services.GetRequiredService<ILoggerFactory>();
var logger = loggerFactory.CreateLogger("...");
lifetime.ApplicationStarted.Register(()
    =>
{
    logger.LogWarning("...");
});

await host.RunAsync();

When HC Function is executed, the ex is thrown, nevertheless HC returns Healthy.

Message: Exception while executing function: Functions.HealthCheck The function invocation context is missing the forwarding task property. 

StackTrace:
Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception System.InvalidOperationException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at Microsoft.Azure.WebJobs.Script.Grpc.DefaultHttpProxyService+<EnsureSuccessfulForwardingAsync>d__6.MoveNext (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.Grpc/Server/DefaultHttpProxyService.cs:52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel+<InvokeResponse>d__103.MoveNext (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs:1099)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:101)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:82)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator+<Coerce>d__3`1.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs:225)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:53)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)

image

Please support.

nnicksize commented 4 months ago

This happens for us too, but only sporadically. According to app insights around 1/10 of the time. It started yesterday at around 14:00 UTC (16:00+0200). We also run on dotnet 8, isolated worker, and on an app service plan with no other services on it.

image

nnicksize commented 4 months ago

@fabiocav No direspect, however your change in patch 4.34.2, commit c7323b4, has changes in the forwarding service and the timeline of the error showing up kinda lines up. :) Thanks for looking at it!

Edit: Nevermind, we're running function host 4.34.1.1 which exactly doesn't yet include your fix I mentioned above.

cplankl commented 4 months ago

This happens in our Azure function too.

Running net8.0 in isolated worker. Function host is also 4.34.1.1. StackTrace is exactly the same as in the initial post.

Our health function looks like this. Nothing special in it.

    [Function("Health")]
    public static IActionResult Health(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)]
        HttpRequest req)
    {
        return new StatusCodeResult(StatusCodes.Status200OK);
    }
peterasamoah7 commented 4 months ago

We recently started seeing this in our function app as well. Any solution or reason why this is happening ?

smockr commented 4 months ago

We have also been seeing this for a just under a week now. I have a vague memory of looking at this previously and seeing that 4.34.2 might fix it, but it seems like maybe that was only a change and has not been confirmed as a fix?

If there is a fix, is it possible to get an indication of a rollout timeline? This is a bit of an annoyance for us as it is failing http triggered requests that have otherwise succeeded, causing retries from third parties to those endpoints.

chrisrlewis commented 4 months ago

We are seeing this too, with the most basic 'Ping' function. Sporadically, not for every request.

Runtime version 4.34.1.1 - also would like an indication of when the fix will be rolled out.

patrick-elmore commented 4 months ago

We are seeing the same thing as well. The endpoint we are getting these errors on is a vanilla health check endpoint that just returns 200 response, almost identical to the code @cplankl posted.

Runtime version - 4.34.1.22669 dotnet-isolated .net6.0

perum commented 4 months ago

Also seeing this and it's causing issues for alerting. Roughly 1 failure per hour, but still intermittent. 4.34 runtime, dotnet-isolated, net8.0. Please advise or get in a fix for this...

mtycholaz commented 4 months ago

I'm seeing this problem happen if I add a custom middleware to a basic function.

Code to reproduce

https://github.com/mtycholaz/azure-function-middleware

gsayem commented 4 months ago

Middleware moved to ConfigureFunctionsWebApplication.

var host = new HostBuilder()
    .ConfigureAppConfiguration((_, config) =>
        config
            .AddJsonFile("appsettings.json", true, true)
            .AddDefaultAzureAppConfiguration(...)
            .AddUserSecrets(Assembly.GetExecutingAssembly(), true))
           .ConfigureFunctionsWebApplication(configureWorker =>
            {
               // The Middlewares
                configureWorker.UseMiddleware<CustomMiddleware>();
            })
ChoOo7 commented 4 months ago

@gsayem Does moving the middleware helps to prevent errors ? Thanks

cplankl commented 4 months ago

@gsayem In my code I do not add custom middlewares:

using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var configuration = new ConfigurationBuilder()
    .SetBasePath(Environment.CurrentDirectory)
    .AddEnvironmentVariables()
    .Build();

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();

However, ConfigureFunctionsWebApplication() will probably register middlewares but it is also a recommendation from Microsoft to do so:

This example includes ASP.NET Core integration to improve performance and provide a familiar programming model when your app uses HTTP triggers. If you do not intend to use HTTP triggers, you can replace the call to ConfigureFunctionsWebApplication with a call to ConfigureFunctionsWorkerDefaults. If you do so, you can remove the reference to Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore from your project file. However, for the best performance, even for functions with other trigger types, you should keep the FrameworkReference to ASP.NET Core.

gsayem commented 4 months ago

@gsayem Does moving the middleware helps to prevent errors ? Thanks

I think yes, also my errors are gone when I moved my middleware.

gsayem commented 4 months ago

@gsayem In my code I do not add custom middlewares:

using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var configuration = new ConfigurationBuilder()
    .SetBasePath(Environment.CurrentDirectory)
    .AddEnvironmentVariables()
    .Build();

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();

However, ConfigureFunctionsWebApplication() will probably register middlewares but it is also a recommendation from Microsoft to do so:

This example includes ASP.NET Core integration to improve performance and provide a familiar programming model when your app uses HTTP triggers. If you do not intend to use HTTP triggers, you can replace the call to ConfigureFunctionsWebApplication with a call to ConfigureFunctionsWorkerDefaults. If you do so, you can remove the reference to Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore from your project file. However, for the best performance, even for functions with other trigger types, you should keep the FrameworkReference to ASP.NET Core.

This should work, I tried your example and it's working as expected.

mtycholaz commented 4 months ago

@gsayem I tried out your recommendation. The error has gone away for me now. I don't fully understand the difference between ConfigureFunctionsWorkerDefaults and ConfigureFunctionsWebApplication but that seemed to have fixed it for me.

melissazeti commented 4 months ago

We're having the same issue and making the ConfigureFunctionsWebApplication to ConfigureFunctionsWorkerDefaults change isn't really an option due to the size of the change. Is there any timeline on a fix?

dzejsien commented 4 months ago

@jviau I assume you can support here. Fix for https://github.com/Azure/azure-functions-host/issues/10119 didn't help.

dzejsien commented 3 months ago

At least from my side, the issue has been fixed with no action from my side in the new runtime version 4.34.2.2

cplankl commented 3 months ago

Yes 4.34.2.2 seems to be fine.

I'm running it in a docker container in image mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0-appservice

Is there an easy way to get information about which runtime version is used in which docker image?