Durable Function - dotnet-isolated - Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")

[gui28347]

• Trying to migrate a function running in-process to out-of-process dotnet-isolated 7.

• Same code works perfectly in in-process mode. Causing this exception in dotnet-isolated 7 after migrating the required code to run as dotnet-isolatd.

Stack:

[2023-11-18T04:13:01.793Z] Function 'MainActivityVM', Invocation id 'b81a7a79-437c-4d76-8ae2-64e48fbbb62e': An exception was thrown by the invocation.
[2023-11-18T04:13:01.794Z] Result: Function 'MainActivityVM', Invocation id 'b81a7a79-437c-4d76-8ae2-64e48fbbb62e': An exception was thrown by the invocation.
Exception: System.AggregateException: One or more errors occurred. (RunMain Exception: Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
[2023-11-18T04:13:01.795Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.RaiseEventAsync(String instanceId, String eventName, Object eventPayload, CancellationToken cancellation)
[2023-11-18T04:13:01.795Z]    at TEST99.SampleFunction.RunMain(FunctionContext context, DurableTaskClient dclient, String vmname, String vmtype) in C:\temp\investigate\AzureAutomation_PersonalDOTNET8\TEST99\SampleFunction.cs:line 82)
[2023-11-18T04:13:01.796Z]  ---> System.Exception: RunMain Exception: Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")
[2023-11-18T04:13:01.797Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.RaiseEventAsync(String instanceId, String eventName, Object eventPayload, CancellationToken cancellation)
[2023-11-18T04:13:01.797Z]    at TEST99.SampleFunction.RunMain(FunctionContext context, DurableTaskClient dclient, String vmname, String vmtype) in C:\temp\investigate\AzureAutomation_PersonalDOTNET8\TEST99\SampleFunction.cs:line 82
[2023-11-18T04:13:01.798Z]    at TEST99.SampleFunction.RunMain(FunctionContext context, DurableTaskClient dclient, String vmname, String vmtype) in C:\temp\investigate\AzureAutomation_PersonalDOTNET8\TEST99\SampleFunction.cs:line 93
[2023-11-18T04:13:01.799Z]    --- End of inner exception stack trace ---

 [Function("MainActivityVM")]
 public static async Task<string> RunMain(
 [ActivityTrigger] FunctionContext context,
 [DurableClient] DurableTaskClient dclient, string vmname, string vmtype)
 {
     CustomStatusDurableFunction customStatus = new();
     try
     {
         int i = 0;
         while (true)
         {
             i++;
             customStatus.progressBar = i;
             await dclient.RaiseEventAsync(context.InvocationId, "StatusUpdate", customStatus); // line 82
             System.Threading.Thread.Sleep(2000);
             if (i == 100)
             {
                 break;
             }
         }
         return "OK";
     }
     catch (Exception ex)
     {
         throw new Exception($"RunMain Exception: {ex}"); // line 93
     }
 }

[anandunv]

Facing same issue . working locally and not working in portal

[anandunv]

@gui28347 I got resolved

Please follow the steps mentioned in the below documentation . here

In my case I haven't set the Storage Blob Data Contributor role access .

[jcageman]

Same issue here! await client.PurgeAllInstancesAsync returns the same error for me locally.

[MarcinJuraszek]

Any updates on this? This goes very much against what the document claims when it comes to external events:

If there is no orchestration instance with the specified instance ID, the event message is discarded.

from Handling external events in Durable Functions (Azure Functions)

[MarcinJuraszek]

Looking more into this, I think I tracked it down to this:

        private bool CheckStatusBeforeRaiseEvent
            => this.durableTaskOptions.ThrowStatusExceptionsOnRaiseEvent ?? this.durabilityProvider.CheckStatusBeforeRaiseEvent;

Unless you set ThrowStatusExceptionOnRaiseEvent explicitly, it'll use whatever the durability provider says we should do, which for Azure Storage says you should check here: https://github.com/Azure/azure-functions-durable-extension/blob/d01791972a4b9f439b3e83f8a0e2f3e80a5b1914/src/WebJobs.Extensions.DurableTask/AzureStorageDurabilityProvider.cs#L59

And with that being true the RaiseEventInternalAsync will first check if the instance is up and healthy, and throw if not here: https://github.com/Azure/azure-functions-durable-extension/blob/d01791972a4b9f439b3e83f8a0e2f3e80a5b1914/src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs#L857

It's throwing InvalidOperationException but I guess for isolated gRPC gets in a way and ultimately we see RpcException.

Setting following in hosts.json makes the exception not happen:

{
    "version": "2.0",
    "extensions": {
        "durableTask": {
            "throwStatusExceptionsOnRaiseEvent": false
        }
    }
}

[Fazer01]

We are facing exaclty the same issue.

@MarcinJuraszek I think you are looking in the wrong repo for the DurableClient. For dotnet-isolated you need the DurableTaskClient implementation in the repository where I have posted this issue. (correct me if I'm wrong).

In the previous 'dotnet in process' mode the RaiseEventAsync works like a charm and does the correct thing when a non-existent orchestratorid is used for RaiseEventAsync.

As stated here (IDurableOrchestrationClient)

• ArgumentException when the instance id does not correspond to a valid orchestration instance.
• InvalidOperationException The orchestration instance with the provided instance id is not running.

For the dotnet-isolated mode however, the documentation says it does silently eat the event that is raised, but in practice the RpcException is thrown...

Just verified that "throwStatusExceptionsOnRaiseEvent": false in the host.json will not throw any exception at all...