search

Azure / azure-functions-durable-extension

Durable Task Framework extension for Azure Functions
MIT License
715 stars 271 forks source link

Orchestration Trigger input binding fails with "Orchestration history state was either missing from the input or not a string value" #2804

Open EmilDamsbo opened 6 months ago

EmilDamsbo commented 6 months ago

Description

Running into the uncaught exception "Orchestration history state was either missing from the input or not a string value".

Setup

  1. Have two functions F1 and F2, the first is an HTTP trigger and the other is an orchestrator function
  2. HTTP trigger F1 schedules a new orchestration instance for F2, and adds an additional input object
  3. Before my debugger enters F2, the input binding fails: 
    [2024-04-26T21:46:38.116Z] An error occurred while executing the orchestrator function 'DispatchJob'.
[2024-04-26T21:46:38.117Z] Result: An error occurred while executing the orchestrator function 'DispatchJob'.
Exception: System.InvalidOperationException: Orchestration history state was either missing from the input or not a string value.
[2024-04-26T21:46:38.118Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.RunOrchestrationAsync(FunctionContext context, BindingMetadata triggerBinding, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 55
[2024-04-26T21:46:38.119Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
[2024-04-26T21:46:38.120Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.EnsureSynchronousExecution(FunctionContext functionContext, FunctionExecutionDelegate next, FunctionsOrchestrationContext orchestrationContext) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 81
[2024-04-26T21:46:38.121Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.RunAsync(TaskOrchestrationContext context, Object input) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 51Stack:    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.RunOrchestrationAsync(FunctionContext context, BindingMetadata triggerBinding, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 55
[2024-04-26T21:46:38.122Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
[2024-04-26T21:46:38.123Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.EnsureSynchronousExecution(FunctionContext functionContext, FunctionExecutionDelegate next, FunctionsOrchestrationContext orchestrationContext) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 81
[2024-04-26T21:46:38.124Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.RunAsync(TaskOrchestrationContext context, Object input) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 51.
[2024-04-26T21:46:38.676Z] Executed 'Functions.DispatchJob' (Failed, Id=56e78500-0d6f-4448-880e-47e1b1d9571a, Duration=7706ms)
[2024-04-26T21:46:38.678Z] System.Private.CoreLib: Exception while executing function: Functions.DispatchJob. Microsoft.Azure.WebJobs.Extensions.DurableTask: Orchestration history state was either missing from the input or not a string value.
[2024-04-26T21:46:38.681Z] c663be9b1bde4b788c83378174a3bb1c: Function 'DispatchJob (Orchestrator)' failed with an error. Reason: Orchestration history state was either missing from the input or not a string value.. IsReplay: False. State: Failed. RuntimeStatus: Failed. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 5. TaskEventId: -1
  4. I debugged, and it seems the orchestration trigger input binding is somehow binding a full object rather than a string in here: a.. https://github.com/Azure/azure-functions-durable-extension/blob/dev/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs#L49 b. triggerInputData is expected to have a .value that is either null or string, but in my case it's a {Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrationContext}. The whole triggerInputData object after binding: 
    {
"BindingMetadata": {
    "Name": "context",
    "Type": "orchestrationTrigger",
    "Direction": 0
},
"Value": {
    "IsAccessed": false,
    "Name": {
        "Name": "DispatchJob",
        "Version": ""
    },
    "InstanceId": "262ca4d6610f479da9e344d0681be433",
    "CurrentUtcDateTime": "2024-04-26T21:47:53.5989272Z",
    "IsReplaying": false,
    "Parent": null,
    "Entities": {}
}
}

Expected behavior

I expect the orchestration trigger to bind successfully, and start executing my function code.

Actual behavior

Function 'DispatchJob (Orchestrator)' failed with an error. Reason: Orchestration history state was either missing from the input or not a string value.

Relevant source code snippets

// insert code snippet here

Known workarounds

I haven't encountered any workarounds yet.

App Details

Screenshots

image

cgillum commented 6 months ago

This is incredibly strange. Does your app contain any customizations, like middleware, injected services, etc., - i.e. things that would normally be done in the Program.cs file?

I can't think of any reason why the trigger binding value would be an object, and why it would be a fully populated FunctionsOrchestrationContext of all things as shown in your screenshot.

It would be great if you could provide a minimal repro app for us to look at.

EmilDamsbo commented 6 months ago

We have quite a large app built around this, and there is potentially some middlewares at play. I'll work to produce a minimum repro of this within the week. Some additional context would also be that the caller to the orchestrator is an abstract generic class which the entrypoint function inherits.

So something like

public abstract class FunctionBase<TRequest>
{
    protected async Task<IActionResult> DispatchJobFromRequest(TRequest request, DurableTaskClient context)
    {
        [...]
        // using the generated extension method for class-based orchestrators, but this reproed without it too
        // https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-dotnet-isolated-overview#source-generator-and-class-based-activities-and-orchestrations
        await context.ScheduleNewDispatchJobInstanceAsync(request);
        [...]
        return new ObjectResult(...])
    }
}

public class RequestHandler : FunctionBase<MyRequest>
{
    [Function]
    public async Task<IActionResult> HandleRequest([HttpTrigger([...])] HttpRequest request, [DurableClient] DurableTaskClient context)
    {
        MyRequest convertedRequest = [convert HttpRequest to MyRequest somehow]

        return this.DispatchJobFromRequest(convertedRequest, context);
    }
}

And the orchestrator function DispatchJob is defined elsewhere not inheriting from the FunctionBase. This is the source-generated extension for the DurableTaskClient (censored the namespace and part of the object name), and I don't see any reason why it would pass the fully-formed FunctionsOrchestrationContext here. image

EmilDamsbo commented 6 months ago

It seems to be related to inheritance when passing objects as an input parameter to the ScheduleNewOrchestrationInstanceAsync-method, because I now have a not-entirely-minimal repro. I will try to upload it tomorrow once I've removed any traces of company IP.

EmilDamsbo commented 6 months ago

I found the offending line. I had previously followed some of the advice in this thread https://github.com/Azure/azure-functions-durable-extension/issues/2527

So the functions' HostBuilder looked like this:

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication(
        builder => new DurableTaskExtensionStartup().Configure(builder)
    )
    .ConfigureServices(services =>
    {
        services.AddHttpClient();
    })
    .Build();

host.Run();

Now this manual call to DurableTaskExtensionStartup.Configure had fixed some of my source generation issues, however it is also causing some unintended middleware issues unbeknownst to me. So when I removed line 3 there with the explicit DurableTaskExtensionStartup.Configure()-call, I stopped encountering the issue.

Repository here demonstrates the issue. it's about as minimal as it gets, it's based on one of the public templates: https://github.com/EmilDamsbo/OrchestrationInputBindingRepro I recommend adding some kind of mechanism in DurableTaskExtensionStartup to ensure there isn't duplicated middlewares.

cgillum commented 6 months ago

@jviau can you take a look? This seems related to another issue that you've helped users with before.

jviau commented 6 months ago

@EmilDamsbo I imagine what happened is your source generation was eventually fixed, so that line: new DurableTaskExtensionStartup().Configure(builder) was now being implicitly called for you. Having that manual call wound up double registering our types, leading to some undefined behavior.

EmilDamsbo commented 6 months ago

I agree that this is most likely what happened after all my experimentation. Would it be possible to catch this DI double-registration on the durable extensions side or should I open an issue elswhere?

jviau commented 6 months ago

We could do some work on the durable side to make this registration re-enterable.

lilyjma commented 2 months ago

@EmilDamsbo - if you're still seeing the issue, can you provide a latest minimal repro please? We'd like to know what exactly is the trigger of the exception. Thanks!

cgillum commented 1 week ago

Closing since the underlying issue has been fixed.

MarcinJuraszek commented 1 week ago

This issue still exists when using the new IHostApplicationBuilder logic being introduced right now (currently in -preview2 versions of Azure Functions). The documentation says to setup the host like this:

var builder = FunctionsApplication.CreateBuilder(args);

// (...) removed because doesn't seem to matter?

var host = builder.Build();

And later calls for builder.ConfigureFunctionsWebApplication() to be used to enable ASP.NET Core integration:

If you want to use ASP.NET Core integration, call builder.ConfigureFunctionsWebApplication().

However, the moment you add that ConfigureFunctionsWebApplication() call, Durable Framework starts failing with:

Exception: System.InvalidOperationException: Orchestration history state was either missing from the input or not a string value.

To reproduce, simply create a new Azure Functions app with a Durable Function in it, update the SDKs to preview versions and change the host setup from 

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();

to

var hostBuilder = FunctionsApplication.CreateBuilder(args);

hostBuilder.ConfigureFunctionsWebApplication();

var host = hostBuilder.Build();

host.Run();

The versions I've used to reproduce it:

<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="2.0.0-preview4" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.4.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.7" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="2.0.0-preview4" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="2.0.0-preview2" />
cgillum commented 1 week ago

Reactivating per @MarcinJuraszek as it seems like a new regression might have been introduced in some of the current preview nuget packages.

@jviau - any ideas what might be going on here?