search

Azure / azure-functions-durable-extension

Durable Task Framework extension for Azure Functions
MIT License
715 stars 267 forks source link

Durable functions do not seem to work with Azure App Configuration Service #2876

Open chassq opened 1 month ago

chassq commented 1 month ago

Description

We have an in-process Azure Durable Functions project whereby we inject the Azure App Configuration Services and set up configuration for the app. What we are seeing is the durable function not coming out of a running state and the error message:

An invalid asynchronous invocation was detected. This can be caused by awaiting non-durable tasks in an orchestrator function's implementation or by middleware that invokes asynchronous code.

We have isolated this to Azure App Configuration service by creating a separate Durable Azure Function (see attached zip). It runs fine when the Azure App Configuration dependency is not there. However, when we tie it in, we get the failure.

For example, if you comment out line Program.cs line 60 the orchestration goes into a completed state else, it stays in a running state. We think maybe this is due to some async task which the Orchestration is detecting? We could not find an open issue on this or any related docs. FunctionApp1.zip

Would there be a work around for this?

Expected behavior

We would expect the app configuration service background worker tasks to get the configuration data and check a sentinel to not interfere with a durable tasks ability to go to a status of completed.

Actual behavior

The durable task is not completing and stays in a running state.

Source code snippets of note

  "extensions": {
    "durableTask": {
      "maxConcurrentOrchestratorFunctions": 1,
      "maxConcurrentActivityFunctions": 200
    }
  },

Known workarounds

None yet.

App Details

VS 2022 info

Microsoft Visual Studio Enterprise 2022 Version 17.11.0 Preview 2.1 VisualStudio.17.Preview/17.11.0-pre.2.1+35017.193 Microsoft .NET Framework Version 4.8.09032

Installed Version: Enterprise

Visual C++ 2022 00476-80000-00000-AA584 Microsoft Visual C++ 2022

ADL Tools Service Provider 1.0 This package contains services used by Data Lake tools

ASA Service Provider 1.0

ASP.NET and Web Tools 17.11.175.11758 ASP.NET and Web Tools

Azure App Service Tools v3.0.0 17.11.175.11758 Azure App Service Tools v3.0.0

Azure Data Lake Tools for Visual Studio 2.6.5000.0 Microsoft Azure Data Lake Tools for Visual Studio

Azure Functions and Web Jobs Tools 17.11.175.11758 Azure Functions and Web Jobs Tools

Azure Stream Analytics Tools for Visual Studio 2.6.5000.0 Microsoft Azure Stream Analytics Tools for Visual Studio

C# Tools 4.11.0-2.24304.1+e68227ea677b76a3c603bd616f03ea6d952b2458 C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools 1.10 Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Entity Framework Core Power Tools 2.6 Adds useful design-time EF Core DbContext features to the Visual Studio Solution Explorer context menu.

Extensibility Message Bus 1.4.39 (main@e8108eb) Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

GitHub Copilot 0.2.1318.45276 GitHub Copilot is an AI pair programmer that helps you write code faster and with less work.

Microsoft Azure Hive Query Language Service 2.6.5000.0 Language service for Hive query

Microsoft Azure Stream Analytics Language Service 2.6.5000.0 Language service for Azure Stream Analytics

Microsoft Azure Tools for Visual Studio 2.9 Support for Azure Cloud Services projects

Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Mono Debugging for Visual Studio 17.11.0 (3174a70) Support for debugging Mono processes with Visual Studio.

NuGet Package Manager 6.11.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

PowerShell Pro Tools for Visual Studio 1.0 A set of tools for developing and debugging PowerShell scripts and modules in Visual Studio.

Razor (ASP.NET Core) 17.11.2.2427801+3d16cb3fea45e74e52068fe60b9f30fc9a675abd Provides languages services for ASP.NET Core Razor.

SQL Server Data Tools 17.11.15.0 Microsoft SQL Server Data Tools

Test Adapter for Boost.Test 1.0 Enables Visual Studio's testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory.

Test Adapter for Google Test 1.0 Enables Visual Studio's testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory.

ToolWindowHostedEditor 1.0 Hosting json editor into a tool window

TypeScript Tools 17.0.30529.2002 TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools 4.11.0-2.24304.1+e68227ea677b76a3c603bd616f03ea6d952b2458 Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools 17.11.0-beta.24274.5+ac872b5a1751457431f37ad2488e5279f8e80caf Microsoft Visual F# Tools

Visual Studio IntelliCode 2.2 AI-assisted development for Visual Studio.

VisualStudio.DeviceLog 1.0 Information about my package

VisualStudio.Mac 1.0 Mac Extension for Visual Studio

VSPackage Extension 1.0 VSPackage Visual Studio Extension Detailed Info

Xamarin 17.11.0.32 (main@f34bff7) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer 17.11.2.6 (remotes/origin/main@8d202db4e2) Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin.Android SDK 13.2.2.0 (d17-5/45b0e14) Xamarin.Android Reference Assemblies and MSBuild support. Mono: d9a6e87 Java.Interop: xamarin/java.interop/d17-5@149d70fe SQLite: xamarin/sqlite/3.40.1@68c69d8 Xamarin.Android Tools: xamarin/xamarin-android-tools/d17-5@ca1552d

chassq commented 1 month ago

To add a bit of followup. We did test with forcing an orchestration termination and that seemed to get it in an end state. Not sure if this is a good idea or not.

[Function(nameof(TerminateOrchestrationActivity))]
public static async Task TerminateOrchestrationActivity(
    [ActivityTrigger] TerminationMessage terminationMessage,
    [DurableClient] DurableTaskClient client,
    FunctionContext context)
{
    ILogger logger = context.GetLogger(nameof(RunFacilityUserIngest));

    try
    {
        var instance = await client.GetInstanceAsync(terminationMessage.InstanceName, getInputsAndOutputs: true, cancellation: context.CancellationToken);
        if (instance != null)
        {
            if (instance.RuntimeStatus != OrchestrationRuntimeStatus.Completed)
            {
                var terminateInstanceOptions = new TerminateInstanceOptions
                {
                    Recursive = true, //Clean up sub-orchestrations if they exist.
                    Output = terminationMessage.Reason
                };

                await client.TerminateInstanceAsync(terminationMessage.InstanceName, terminateInstanceOptions);
            }
        }
    }
    catch (Exception ex)
    {

        logger.LogError(message: $"{nameof(TerminateOrchestrationActivity)} experienced an exception. Message: {ex.Message} | Details: {ex.StackTrace}");
    }      
}

public class TerminationMessage(string instanceName, string reason)
{
    public string InstanceName { get; private set; } = instanceName;
    public string Reason { get; private set; } = reason;
}
chassq commented 3 weeks ago

Any update on this issue?

lilyjma commented 2 weeks ago

@chassq - apologies for the delay. This is a known issue. Unfortunately we haven't been able to prioritize the fix. Please see if the workaround described here works for you.

chassq commented 2 weeks ago

Thanks for the reply!