search

Azure / azure-functions-durable-extension

Durable Task Framework extension for Azure Functions
MIT License
714 stars 270 forks source link

"An invalid asynchronous invocation was detected" for custom timeout method #2890

Closed AKazaconoksIf closed 1 week ago

AKazaconoksIf commented 2 months ago

What version of .NET does your existing project use?

.NET 6

What version of .NET are you attempting to target?

.NET 8

Description

As Timeout attribute no longer exists in isolated functions, we found here orchestration context usage as an alternative. However time to time we are receiving error:

Exception while executing function: Functions.TimeoutOrchestrationFunction An invalid asynchronous invocation was detected. This can be caused by awaiting non-durable tasks in an orchestrator function's implementation or by middleware that invokes asynchronous code.

Our code:

using Microsoft.Extensions.Logging;
using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;

namespace Abc;

public class AbcFunctions(
    IFeatureToggleProvider featureToggleProvider,
    ILogger<AbcFunctions> log)
{
    private const string OrchestrationFunction = "TimeoutOrchestrationFunction";

    [Function("Abc_Timer")]
    public static async Task RunTimer(
        [TimerTrigger("0 0 10 * * *")] TimerInfo myTimer,
        [DurableClient] DurableTaskClient starter,
        CancellationToken cancellationToken
    )
    {
        await starter.ScheduleNewOrchestrationInstanceAsync(OrchestrationFunction, cancellationToken);
    }

    [Function(OrchestrationFunction)]
    public static async Task<bool> Run([OrchestrationTrigger] TaskOrchestrationContext context)
    {
        var date = context.GetInput<DateTime?>();

        TimeSpan timeout = TimeSpan.FromHours(2);
        DateTime deadline = context.CurrentUtcDateTime.Add(timeout);

        using var cts = new CancellationTokenSource();
        Task activityTask = context.CallActivityAsync("ExecuteFunction", (date));
        Task timeoutTask = context.CreateTimer(deadline, cts.Token);
        Task winner = await Task.WhenAny(activityTask, timeoutTask);
        if (winner == activityTask)
        {
            cts.Cancel();
            return true;
        }
        return false;
    }

    [Function("ExecuteFunction")]
    public bool ExecuteFunction([ActivityTrigger] DateTime? date)
    {
        //function logic
    }
}

Full exception:

Message: Exception while executing function: Functions.TimeoutOrchestrationFunction An invalid asynchronous invocation was detected. This can be caused by awaiting non-durable tasks in an orchestrator function's implementation or by middleware that invokes asynchronous code. 
Exception type: DurableTask.Core.Exceptions.OrchestrationFailureException
Call Stack: 
Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception DurableTask.Core.Exceptions.OrchestrationFailureException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.OutOfProcMiddleware+<>c__DisplayClass10_0+<<CallOrchestratorAsync>b__0>d.MoveNext (Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\OutOfProcMiddleware.cs:145)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor`1+<>c__DisplayClass7_0+<<TryExecuteAsync>b__0>d.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\TriggeredFunctionExecutor.cs:51)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)

Project configuration and dependencies

No response

Link to a repository that reproduces the issue

No response

joaocpribeiro commented 2 weeks ago

I was having the same issue. I believe that the problem is that you are awaiting for CreateTimer, which is not an activity. It seems that, in orchestrator, we can only await for activity functions.

cgillum commented 2 weeks ago

I believe that the problem is that you are awaiting for CreateTimer, which is not an activity. It seems that, in orchestrator, we can only await for activity functions.

This is not accurate. Calls to CreateTimer can also be awaited - not just activity calls. You're allowed to await any call to a Task that is returned by any of the context methods.

cgillum commented 2 weeks ago

@AKazaconoksIf your code looks fine. In cases like this, it's often some custom middleware that's injecting an illegal await call into the orchestrator function's execution path. Do you have any custom middleware set up in your app?

jmelosegui commented 2 weeks ago

This issue might be related.

An invalid asynchronous invocation was detected

AKazaconoksIf commented 1 week ago

Issue was related to the code which is executed inside ExecuteFunction as the function was timeouting before the timeout from orchestrator occurred. The only thing helped was to increase overall timeout value for all functions, which was not the solution we wanted with this custom timeout setup, so for now seems like there is no real alternative for Timeout attribute from in-process functions, unless it was created within last few months.