Grpc Error in `dotnet-isolated` in net7 and docker

[justinmchase]

I have two functions, one is a CronTrigger, which appears to be working fine the other is an OrchestrationTrigger function. This is running in dotnet-isolated, net7, in docker.

I can see in my logs that the cron is launching, it also successfully schedules the other function, i can see the instanceId in the logs.

[2022-12-17T07:00:04.364Z] I "Start: 3fdccdf3-be99-40e3-95c8-53cf6c590910 Microsoft.Azure.Functions.Worker.TimerInfo" {"category":"Recovery.Functions.HelloProviders"}
[2022-12-17T07:00:04.383Z] Start: 3fdccdf3-be99-40e3-95c8-53cf6c590910 Microsoft.Azure.Functions.Worker.TimerInfo
[2022-12-17T07:00:04.616Z] I "Started: 5c832cad72f1403ead887f06e6456f8a..." {"category":"Recovery.Functions.HelloProviders"}
[2022-12-17T07:00:04.617Z] Started: 5c832cad72f1403ead887f06e6456f8a...
[2022-12-17T07:00:04.728Z] Executed 'Functions.CronTest' (Failed, Id=2e972014-43a1-479e-8b10-a412039749a2, Duration=6640ms)
[2022-12-17T07:00:04.728Z] System.Private.CoreLib: Exception while executing function: Functions.CronTest. System.Private.CoreLib: Result: Failure
...

The line where I call WaitForInstanceCompletionAsync(instanceId, ...) is where it throws an exception. Here are the details:

{
    "created": "@1671260404.629909577",
    "description": "Error received from peer ipv4:127.0.0.1:4001",
    "file": "/var/local/git/grpc/src/core/lib/surface/call.cc",
    "file_line": 1070,
    "grpc_message": "Exception was thrown by handler.",
    "grpc_status": 2
}

I'm not sure how this can happen. This is running in a pod, which has both processes...

Example code

using System.Threading;
using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;

namespace Recovery.Functions;

public class HelloExample
{
  private readonly ILogger logger;

  public HelloExample(ILogger<HelloExample> logger) => this.logger = logger;

  [Function(nameof(HelloExample))]
  public async Task<string> Hello([OrchestrationTrigger] TaskOrchestrationContext context, string name)
  {
    var instanceId = context.InstanceId;
    this.logger.LogInformation("Hello Workflow Running: {name} ({instanceId})", name, instanceId);

    return await Task.FromResult($"Hello {name}!");
  }

  [Function(nameof(CronTest))]
  public async Task CronTest(
    [TimerTrigger("* * * * * *")] TimerInfo timerInfo,
    [DurableClient] DurableClientContext durableContext,
    FunctionContext context
  )
  {
    this.logger.Info($"Start: {context.FunctionId} {timerInfo}");
    var instanceId = await durableContext.Client.ScheduleNewOrchestrationInstanceAsync(nameof(HelloExample), "Justin");
    this.logger.LogInformation("Started: {instanceId}...", instanceId);

    var result = await durableContext.Client.WaitForInstanceCompletionAsync(instanceId, CancellationToken.None, true);
    this.logger.LogInformation("Result: {SerializedOutput}", result.SerializedOutput);
    await Task.CompletedTask;
  }
}

Full error and stack

System.Private.CoreLib: Exception while executing function: Functions.CronTest. System.Private.CoreLib: Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1671260404.629909577","description":"Error received from peer ipv4:127.0.0.1:4001","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Exception was thrown by handler.","grpc_status":2}"))
---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1671260404.629909577","description":"Error received from peer ipv4:127.0.0.1:4001","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Exception was thrown by handler.","grpc_status":2}")
at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.WaitForInstanceCompletionAsync(String instanceId, CancellationToken cancellationToken, Boolean getInputsAndOutputs)
at Recovery.Functions.HelloExamples.CronTest(TimerInfo timerInfo, DurableClientContext durableContext, FunctionContext context) in /var/recovery/apps/recovery-provider-api/src/recovery-provider-api/Functions/HelloExamples.cs:line 33
at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 18
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82
Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 18
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82.

[justinmchase]

The orchestration function itself seems to be running fine:

Executing 'Functions.HelloExample' (Reason='(null)', Id=c282b3cd-430a-401d-8f32-054076109771)
I "Hello Workflow Running: (null) (17c9b0b5037c4b9e81730d626715a0a1)" {"category":"Recovery.Functions.HelloExample"}
Hello Workflow Running: (null) (17c9b0b5037c4b9e81730d626715a0a1)
Executed 'Functions.HelloExample' (Succeeded, Id=c282b3cd-430a-401d-8f32-054076109771, Duration=1ms)
17c9b0b5037c4b9e81730d626715a0a1: Function 'HelloExample (Orchestrator)' completed. ContinuedAsNew: False. IsReplay: False. Output: (36 bytes). State: Completed. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.9.0. SequenceNumber: 118. TaskEventId: -1

[i02coroj]

Hi. Same issue as @justinmchase here. It happens to me even when creating a new Durable Function using the template from VS:

Any update on it, please?

[justinmchase]

I can't remember for sure but I think it was adding the AzureWebJobsSecretStorageType to the local.settings.json file that solved it for me.

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "<redacted>",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    "AzureWebJobsSecretStorageType": "files"
  },
}

[i02coroj]

Mmm, thanks for that @justinmchase, but no luck, that didn't make the trick for me. Same error.

[i02coroj]

But actually you pointed to the right direction @justinmchase , it was an storage issue. For some reason it's not working for me when running locally using Azurite (even when Azurite is working ok for other APIs). I just shut down my Azurite container and run the storage emulator instead and started to work. It's a pain, as using Azurite with Docker is much easier, actually I think Storage Emulator is deprecated? But it works now anyway.

Thanks!

[justinmchase]

Check the version, now that you say it I recall having to update the azurite version I was using also.

[stukalin]

Same here. Have tried to upgrade azurite and the AzureWebJobsSecretStorageType - didn't help

[todosrc]

I have same issue , how to ?

[todosrc]

i have resolved the issue need register on program.cs

  services.AddDurableTaskClient(builder =>
        {
            // Configure options for this builder. Can be omitted if no options customization is needed.
            builder.Configure(opt => { });
            builder.UseGrpc(); // multiple overloads available for providing gRPC information

            // AddDurableTaskClient allows for multiple named clients by passing in a name as the first argument.
            // When using a non-default named client, you will need to make this call below to have the
            // DurableTaskClient added directly to the DI container. Otherwise IDurableTaskClientProvider must be used
            // to retrieve DurableTaskClients by name from the DI container. In this case, we are using the default
            // name, so the line below is NOT required as it was already called for us.
            builder.RegisterDirectly();
        });

[ctrl-brk]

It seems like we soon will celebrate a one year anniversary of this issue. None of the above solutions help me. Still getting this stupid error with no details whatsoever. If I switch to .NET 6 - everything works fine.

[manishyaadav]

same error, in local it's working fine but in when deploying to docker where my azurite is running getting grpc error when creating durable function