search

microsoft / durabletask-dotnet

Out-of-process .NET SDK for the Durable Task Framework
MIT License
114 stars 33 forks source link

Grpc Error calling WaitForInstanceCompletionAsync in dotnet-isolated #143

Open rob-baily opened 1 year ago

rob-baily commented 1 year ago

I actually submitted this first at https://github.com/Azure/azure-functions-core-tools/issues/3227 but some other research indicated this might be the correct place.

The code below is taken from the MS provided template and I added the await client.WaitForInstanceCompletionAsync call which is where the error is triggered. It is should be easy to reproduce since it is basically the MS template with one line added. This is using .NET 7.0 in an isolated process. I upgraded to the latest packages so here are my versions:

    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.13.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.0.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.0.13" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.9.0" />

I have also seen the same behavior is .NET 6.0 isolated as well as noted in the other issue.

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
using Microsoft.Extensions.Logging;

namespace DurableFunctionExample
{
    public static class Function
    {
        [Function(nameof(Function))]
        public static async Task<List<string>> RunOrchestrator(
            [OrchestrationTrigger] TaskOrchestrationContext context)
        {
            ILogger logger = context.CreateReplaySafeLogger(nameof(Function));
            logger.LogInformation("Saying hello.");
            var outputs = new List<string>();

            // Replace name and input with values relevant for your Durable Functions Activity
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "Tokyo"));
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "Seattle"));
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "London"));

            // returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
            return outputs;
        }

        [Function(nameof(SayHello))]
        public static string SayHello([ActivityTrigger] string name, FunctionContext executionContext)
        {
            ILogger logger = executionContext.GetLogger("SayHello");
            logger.LogInformation("Saying hello to {name}.", name);
            return $"Hello {name}!";
        }

        [Function("Function_HttpStart")]
        public static async Task<HttpResponseData> HttpStart(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
            [DurableClient] DurableTaskClient client,
            FunctionContext executionContext)
        {
            ILogger logger = executionContext.GetLogger("Function_HttpStart");

            // Function input comes from the request content.
            string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
                nameof(Function));

            logger.LogInformation("Started orchestration with ID = '{instanceId}'.", instanceId);

            var metaData = await client.WaitForInstanceCompletionAsync(instanceId);
            // Returns an HTTP 202 response with an instance management payload.
            // See https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-http-api#start-orchestration
            return client.CreateCheckStatusResponse(req, instanceId);
        }
    }
}

The full stack is:

[2023-04-12T17:49:49.217Z] Function 'Function_HttpStart', Invocation id 'ddbe2542-1404-45af-b96b-ec9762a28dd2': An exception was thrown by the invocation.
[2023-04-12T17:49:49.221Z] Result: Function 'Function_HttpStart', Invocation id 'ddbe2542-1404-45af-b96b-ec9762a28dd2': An exception was thrown by the invocation.
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler."))
[2023-04-12T17:49:49.222Z]  ---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
[2023-04-12T17:49:49.224Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.WaitForInstanceCompletionAsync(String instanceId, Boolean getInputsAndOutputs, CancellationToken cancellation)
[2023-04-12T17:49:49.225Z]    at DurableFunctionExample.Function.HttpStart(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\RobBaily\source\repos\DurableFunctionExample\Function.cs:line 50
[2023-04-12T17:49:49.227Z]    --- End of inner exception stack trace ---
[2023-04-12T17:49:49.228Z]    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-04-12T17:49:49.230Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-04-12T17:49:49.232Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-04-12T17:49:49.234Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-04-12T17:49:49.236Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.237Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.237Z] Executed 'Functions.Function_HttpStart' (Failed, Id=ddbe2542-1404-45af-b96b-ec9762a28dd2, Duration=360ms)
[2023-04-12T17:49:49.240Z] System.Private.CoreLib: Exception while executing function: Functions.Function_HttpStart. System.Private.CoreLib: Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler."))
[2023-04-12T17:49:49.238Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.241Z]  ---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
[2023-04-12T17:49:49.243Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.WaitForInstanceCompletionAsync(String instanceId, Boolean getInputsAndOutputs, CancellationToken cancellation)
[2023-04-12T17:49:49.242Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-04-12T17:49:49.247Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.245Z]    at DurableFunctionExample.Function.HttpStart(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\RobBaily\source\repos\DurableFunctionExample\Function.cs:line 50
[2023-04-12T17:49:49.248Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
[2023-04-12T17:49:49.251Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-04-12T17:49:49.253Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-04-12T17:49:49.256Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
Stack:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-04-12T17:49:49.250Z]    --- End of inner exception stack trace ---
[2023-04-12T17:49:49.257Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-04-12T17:49:49.259Z]    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-04-12T17:49:49.260Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-04-12T17:49:49.262Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-04-12T17:49:49.263Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-04-12T17:49:49.266Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.267Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.269Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.270Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-04-12T17:49:49.271Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.272Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
[2023-04-12T17:49:49.267Z] Executing 'Functions.Function' (Reason='(null)', Id=8c59ab82-641b-45e7-b164-d8dbb023820b)
[2023-04-12T17:49:49.274Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-04-12T17:49:49.277Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-04-12T17:49:49.264Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-04-12T17:49:49.278Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77.
[2023-04-12T17:49:49.280Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-04-12T17:49:49.282Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.284Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.285Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.286Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-04-12T17:49:49.288Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.289Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
[2023-04-12T17:49:49.291Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-04-12T17:49:49.293Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-04-12T17:49:49.294Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
[2023-04-12T17:49:49.295Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request, WorkerOptions workerOptions) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 84
Stack:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-04-12T17:49:49.296Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-04-12T17:49:49.297Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-04-12T17:49:49.298Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-04-12T17:49:49.300Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.301Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.302Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-04-12T17:49:49.304Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-04-12T17:49:49.306Z] --- End of stack trace from previous location ---
[2023-04-12T17:49:49.307Z] Saying hello.
[2023-04-12T17:49:49.308Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
[2023-04-12T17:49:49.310Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-04-12T17:49:49.311Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-04-12T17:49:49.312Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
[2023-04-12T17:49:49.313Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request, WorkerOptions workerOptions) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 84.
galvesribeiro commented 1 year ago

I have the same problem but in my case, the exception happen when iterating in the result of the instance query response:

var instancesQuery = client.GetAllInstancesAsync(new OrchestrationQuery(Statuses: _runningStatuses));
            await foreach (var instance in instancesQuery){ ... }

The await on the foreach throws this:

image

I can't use the storage emulator as others implied since I'm on MacOS so I'm stuck with Azurite.

Any clue would be appreciated.

jviau commented 1 year ago

@galvesribeiro did you follow #148 for running on MacOS?

galvesribeiro commented 1 year ago

I did. It works only with the default Azure Tables backend. If I use Netherite or SQL backends, the error is back again.

jviau commented 1 year ago

When using either of those backends, have you added the corresponding worker package?

https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.Extensions.DurableTask.Netherite https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.Extensions.DurableTask.SqlServer

It is necessary to add those packages as it informs the host to add the corresponding host extension.

I realize this may not be documented just yet.

galvesribeiro commented 1 year ago

@jviau yep, I added the isolated packages. I see the data being written on either the storage and sql respectively and if I remove the call to client.GetAllInstancesAsync everything else just works.

davidmrdavid commented 1 year ago

@galvesribeiro: just to make sure I understand your situation. Is the following summary accurate?

First, you were experiencing this error across all storage providers / backends. Then, you followed the guidance here (https://github.com/microsoft/durabletask-dotnet/issues/148) and that solved it for the Azure Storage backend. And now, you only experience this error in the MSSQL and Netherite backends. Is that correct?

galvesribeiro commented 1 year ago

Yep. The native gRPC issue with Apple Silicon was actually not the problem. It was masking the real problem.

Badabum commented 1 year ago

Also getting the same grpc error when calling client.GetAllInstancesAsync() using Netherite backend. GetInstanceAsync() works fine though.

Schmaga commented 1 year ago

We are encountering the same Grpc Error, and it happens on Windows, Unix, and Mac. We receive essentially the same stack trace, when calling WaitForCompletionAsync, but also with PurgeInstancesAsync.

source-studio commented 1 year ago

I'm also experiencing the same issue when calling DurableTaskClient.GetAllInstancesAsync.

Environment: MacBook M1 Pro, Azurite, Netherite storage provider

I've tried manipulating the OrchestrationQuery parameters, but to no avail. Other operations, such as PurgeInstancesAsync do succeed, but only when the createdFrom and createdTo parameters are specified - leaving these as default, results in a similar error.

The exception is quite generic, presumably hiding the underlying failure, so it's hard to diagnose:

Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
   at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.<>c__DisplayClass15_0.<<GetAllInstancesAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.DurableTask.Pageable.FuncAsyncPageable`1.AsPagesCore(String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellation)+MoveNext()
   at Microsoft.DurableTask.Pageable.FuncAsyncPageable`1.AsPagesCore(String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellation)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
davidmrdavid commented 1 year ago

I can't comment on the Netherite issue yet, but at least for the default Azure Storage backend, I believe this error was coming from a null-reference exception that we recently patched here: https://github.com/Azure/durabletask/pull/910. Note that this has not been released yet.

I'm taking a note to investigate the Netherite equivalent.

vandenbergjp commented 1 year ago

I had the same exception when trying to run it locally. It turned out that I did not have a local.settings.json file (I thought I did not need it because I had the same values in my launchsettings.json). After adding a file with the following content, it worked as expected.

{ "IsEncrypted": false, "Values": { "AzureWebJobsSecretStorageType": "files", "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated" } }

Maybe this helps.

vandenbergjp commented 1 year ago

Maybe this helps other people that encounter this error message. After solving this locally, I had the same error message when I deployed it to an Azure Function App. It took me a couple of days, but I finally figured out that the durable functions use table storage. My Azure Function App only had permission to blobs and queues, but not to Table Storage (The account that I used to locally test it, had those permissions).

Maybe the documentation can make it clearer that the function app requires these permissions.

Also, maybe the exception that is thrown can make it clearer that the app has insufficient permissions, because now I got the very generic message that does not give any clues.

Result: Failure Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")) ---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.") at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.GetInstancesAsync(String instanceId, Boolean getInputsAndOutputs, CancellationToken cancellation) at Lias.Dataloader.FunctionApp.CsvZipFileUploadedTrigger.RunAsync(String myBlob, DurableTaskClient durableTaskClient, Guid tenant, String name) in /src/Lias.Dataloader.FunctionApp/CsvZipFileUploadedTrigger.cs:line 65 at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22 --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker2.<>c.<InvokeAsync>b__6_0(Task1 t) in D:\a_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32 at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22 at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77 at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88 Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker2.<>c.<InvokeAsync>b__6_0(Task1 t) in D:\a_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32 at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22 at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77 at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

syazdian commented 11 months ago

Maybe this helps other people that encounter this error message. After solving this locally, I had the same error message when I deployed it to an Azure Function App. It took me a couple of days, but I finally figured out that the durable functions use table storage. My Azure Function App only had permission to blobs and queues, but not to Table Storage (The account that I used to locally test it, had those permissions).

Maybe the documentation can make it clearer that the function app requires these permissions.

Also, maybe the exception that is thrown can make it clearer that the app has insufficient permissions, because now I got the very generic message that does not give any clues.

Result: Failure Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")) ---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.") at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.GetInstancesAsync(String instanceId, Boolean getInputsAndOutputs, CancellationToken cancellation) at Lias.Dataloader.FunctionApp.CsvZipFileUploadedTrigger.RunAsync(String myBlob, DurableTaskClient durableTaskClient, Guid tenant, String name) in /src/Lias.Dataloader.FunctionApp/CsvZipFileUploadedTrigger.cs:line 65 at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22 --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker2.<>c.<InvokeAsync>b__6_0(Task1 t) in D:\awork\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32 at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in //src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22 at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77 at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88 Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker2.<>c.b__6_0(Task1 t) in D:\a_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32 at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\awork\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in //src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22 at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77 at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

I should say Thank You. We had the same problem, luckily I found your solution. It's Dec 2023 and the problem has not solved yet!

jcageman commented 10 months ago

Also running into this issue. I created a clean .net 8 isolated function project upgrade to latest nuget versions and getting the exact same error locally. If anyone knows a workaround please share :)

        [Function("HttpTriggerTest")]
        public async Task<HttpResponseData> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req, [DurableClient] DurableTaskClient client)
        {
            _logger.LogInformation("C# HTTP trigger function processed a request.");

            try
            {
                var result = await client.PurgeAllInstancesAsync(new PurgeInstancesFilter(null, DateTime.Today.AddDays(-30),
                    new List<OrchestrationRuntimeStatus>
                    {
                        OrchestrationRuntimeStatus.Completed,
                        OrchestrationRuntimeStatus.Failed,
                        OrchestrationRuntimeStatus.Terminated
                    })
                );

                _logger.LogInformation(
                    "{FunctionName} result at {Date}: {PurgedInstanceCount} records were deleted.",
                    nameof(TimedTrigger), DateTime.Now, result.PurgedInstanceCount);
            }
            catch (Exception e)
            {
                _logger.LogError(e, "{FunctionName}: Failed to clean instance history data",
                    nameof(TimedTrigger));
            }

            var response = req.CreateResponse(HttpStatusCode.OK);
            response.Headers.Add("Content-Type", "text/plain; charset=utf-8");

            response.WriteString("Welcome to Azure Functions!");

            return response;
        }

client.PurgeAllInstancesAsync is throwing: "Failed to clean instance history data Exception: Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")"

RobARichardson commented 10 months ago

We're also running into this same issue when using Netherite. We use WaitForInstanceCompletionAsync to implement Singleton Orchestrators. We were hoping to switch from Azure Storage to Netherite but unfortunately this is a deal breaker.

Exception: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.CreateMetadata(OrchestrationState state, Boolean includeInputsAndOutputs) at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.WaitForInstanceCompletionAsync(String instanceId, Boolean getInputsAndOutputs, CancellationToken cancellation)
prashantagrawalcrowe commented 8 months ago

Also running into this issue. I created a clean .net 8 isolated function project upgrade to latest nuget versions and getting the exact same error locally. If anyone knows a workaround please share :)

        [Function("HttpTriggerTest")]
        public async Task<HttpResponseData> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req, [DurableClient] DurableTaskClient client)
        {
            _logger.LogInformation("C# HTTP trigger function processed a request.");

            try
            {
                var result = await client.PurgeAllInstancesAsync(new PurgeInstancesFilter(null, DateTime.Today.AddDays(-30),
                    new List<OrchestrationRuntimeStatus>
                    {
                        OrchestrationRuntimeStatus.Completed,
                        OrchestrationRuntimeStatus.Failed,
                        OrchestrationRuntimeStatus.Terminated
                    })
                );

                _logger.LogInformation(
                    "{FunctionName} result at {Date}: {PurgedInstanceCount} records were deleted.",
                    nameof(TimedTrigger), DateTime.Now, result.PurgedInstanceCount);
            }
            catch (Exception e)
            {
                _logger.LogError(e, "{FunctionName}: Failed to clean instance history data",
                    nameof(TimedTrigger));
            }

            var response = req.CreateResponse(HttpStatusCode.OK);
            response.Headers.Add("Content-Type", "text/plain; charset=utf-8");

            response.WriteString("Welcome to Azure Functions!");

            return response;
        }

client.PurgeAllInstancesAsync is throwing: "Failed to clean instance history data Exception: Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")"

Even I am facing the same issue, any workaround or solution for this problem?

jcageman commented 8 months ago

It seems its actually not supported checking the code: https://github.com/microsoft/durabletask-dotnet/blob/4de17486f97ce4507ecbd496cf877faf7e4cf78a/src/Client/Core/DurableTaskClient.cs#L399

MarioTiscareno commented 7 months ago

I am also interested in a work around.

jcageman commented 7 months ago

Also running into this issue. I created a clean .net 8 isolated function project upgrade to latest nuget versions and getting the exact same error locally. If anyone knows a workaround please share :)

        [Function("HttpTriggerTest")]
        public async Task<HttpResponseData> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req, [DurableClient] DurableTaskClient client)
        {
            _logger.LogInformation("C# HTTP trigger function processed a request.");

            try
            {
                var result = await client.PurgeAllInstancesAsync(new PurgeInstancesFilter(null, DateTime.Today.AddDays(-30),
                    new List<OrchestrationRuntimeStatus>
                    {
                        OrchestrationRuntimeStatus.Completed,
                        OrchestrationRuntimeStatus.Failed,
                        OrchestrationRuntimeStatus.Terminated
                    })
                );

                _logger.LogInformation(
                    "{FunctionName} result at {Date}: {PurgedInstanceCount} records were deleted.",
                    nameof(TimedTrigger), DateTime.Now, result.PurgedInstanceCount);
            }
            catch (Exception e)
            {
                _logger.LogError(e, "{FunctionName}: Failed to clean instance history data",
                    nameof(TimedTrigger));
            }

            var response = req.CreateResponse(HttpStatusCode.OK);
            response.Headers.Add("Content-Type", "text/plain; charset=utf-8");

            response.WriteString("Welcome to Azure Functions!");

            return response;
        }

client.PurgeAllInstancesAsync is throwing: "Failed to clean instance history data Exception: Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")"

Even I am facing the same issue, any workaround or solution for this problem?

workaround here is not passing null, but a from date like 30 days ago or so. It seems null / datetime.minvalue all don't work somehow.

talhanaveedcoop commented 7 months ago

Facing the same issue when I try to get all the instances

var instances = await client.GetAllInstancesAsync(new OrchestrationQuery
{
    CreatedFrom = DateTimeOffset.Now.AddDays(-2),
    CreatedTo = DateTimeOffset.Now,
    Statuses = new List<OrchestrationRuntimeStatus>
{
    OrchestrationRuntimeStatus.Running,
    OrchestrationRuntimeStatus.Pending,
}
    }).ToListAsync();

image

[2024-04-11T13:53:02.177Z] [15:53:02 ERR] Error in bong trigger: Status(StatusCode="Unknown", Detail="Exception was thrown by handler."), Message: Status(StatusCode="Unknown", Detail="Exception was thrown by handler."), InnerException: null
[2024-04-11T13:53:02.179Z] Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
[2024-04-11T13:53:02.180Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.<>c__DisplayClass18_0.<<GetAllInstancesAsync>b__0>d.MoveNext()
[2024-04-11T13:53:02.181Z] --- End of stack trace from previous location ---
[2024-04-11T13:53:02.183Z]    at Microsoft.DurableTask.Pageable.FuncAsyncPageable`1.AsPagesCore(String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellation)+MoveNext()
[2024-04-11T13:53:02.183Z]    at Microsoft.DurableTask.Pageable.FuncAsyncPageable`1.AsPagesCore(String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellation)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
[2024-04-11T13:53:02.185Z]    at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
[2024-04-11T13:53:02.186Z]    at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
[2024-04-11T13:53:02.187Z]    at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
[2024-04-11T13:53:02.188Z]    at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
[2024-04-11T13:53:02.189Z]    at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
[2024-04-11T13:53:02.191Z]    at CoopNorge.Bongs.Trigger.Functions.BongTrigger.Run(TimerInfo myTimer, DurableTaskClient client) in C:\Dev\Projects\BongsIkano\solutions\CoopNorge.Bongs\CoopNorge.Bongs\CoopNorge.Bongs.Trigger\Functions\BongTrigger.cs:line 30
johnthom commented 6 months ago

Temporary solution...

Experienced exactly the same issue. Because this was working for me earlier, I was able to determine that this started happening after updating package "Microsoft.Azure.Functions.Worker.Extensions.DurableTask" from 1.1.1 to 1.1.2 (latest). After backing this off and going back to 1.1.1 the issue went away.

Facing the same issue when I try to get all the instances

var instances = await client.GetAllInstancesAsync(new OrchestrationQuery
{
    CreatedFrom = DateTimeOffset.Now.AddDays(-2),
    CreatedTo = DateTimeOffset.Now,
    Statuses = new List<OrchestrationRuntimeStatus>
{
    OrchestrationRuntimeStatus.Running,
    OrchestrationRuntimeStatus.Pending,
}
    }).ToListAsync();

image

[2024-04-11T13:53:02.177Z] [15:53:02 ERR] Error in bong trigger: Status(StatusCode="Unknown", Detail="Exception was thrown by handler."), Message: Status(StatusCode="Unknown", Detail="Exception was thrown by handler."), InnerException: null
[2024-04-11T13:53:02.179Z] Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
[2024-04-11T13:53:02.180Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.<>c__DisplayClass18_0.<<GetAllInstancesAsync>b__0>d.MoveNext()
[2024-04-11T13:53:02.181Z] --- End of stack trace from previous location ---
[2024-04-11T13:53:02.183Z]    at Microsoft.DurableTask.Pageable.FuncAsyncPageable`1.AsPagesCore(String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellation)+MoveNext()
[2024-04-11T13:53:02.183Z]    at Microsoft.DurableTask.Pageable.FuncAsyncPageable`1.AsPagesCore(String continuationToken, Nullable`1 pageSizeHint, CancellationToken cancellation)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
[2024-04-11T13:53:02.185Z]    at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
[2024-04-11T13:53:02.186Z]    at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
[2024-04-11T13:53:02.187Z]    at Microsoft.DurableTask.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
[2024-04-11T13:53:02.188Z]    at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
[2024-04-11T13:53:02.189Z]    at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
[2024-04-11T13:53:02.191Z]    at CoopNorge.Bongs.Trigger.Functions.BongTrigger.Run(TimerInfo myTimer, DurableTaskClient client) in C:\Dev\Projects\BongsIkano\solutions\CoopNorge.Bongs\CoopNorge.Bongs\CoopNorge.Bongs.Trigger\Functions\BongTrigger.cs:line 30
anpost-stephenomalley commented 1 month ago

Is there any plans to fix this issue in the near future?

I see this issue now using "Microsoft.Azure.Functions.Worker.Extensions.DurableTask" version 1.1.2 and I'm hoping changing to use 1.1.1 works.