Getting io.grpc.StatusRuntimeException: UNKNOWN when calling durableClient.waitForInstanceCompletion

[kushals14]

Using v1.1.0 for durabletask-client and durabletask-azure-functions, Using host function v4.21.1 and extension bundle 4.5.0.

Instance creation and orchestration handling works fine. There is no issue using durableContext.createCheckStatusResponse(request, instanceId) after instance creation, the issue only exists while calling client.waitForInstanceCompletion(instanceId, Duration.ofSeconds(timeoutInSeconds), true)

I am also using this in conjunction with Orchestration waitForExternalEvent and eventGridTrigger to raiseEvent on the created instance.

Exception stack trace -

Exception: StatusRuntimeException: UNKNOWN: Exception was thrown by handler.
Stack: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
    at com.microsoft.azure.functions.worker.broker.EnhancedJavaMethodExecutorImpl.execute(EnhancedJavaMethodExecutorImpl.java:22)
    at com.microsoft.azure.functions.worker.chain.FunctionExecutionMiddleware.invoke(FunctionExecutionMiddleware.java:19)
    at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21)
    at com.microsoft.durabletask.azurefunctions.internal.middleware.OrchestrationMiddleware.invoke(OrchestrationMiddleware.java:29)
    at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21)
    at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:125)
    at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:34)
    at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
    at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:44)
    at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:94)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: Exception was thrown by handler.
    at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
    at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
    at com.microsoft.durabletask.implementation.protobuf.TaskHubSidecarServiceGrpc$TaskHubSidecarServiceBlockingStub.waitForInstanceCompletion(TaskHubSidecarServiceGrpc.java:1130)
    at com.microsoft.durabletask.DurableTaskGrpcClient.waitForInstanceCompletion(DurableTaskGrpcClient.java:189)
    ... 20 more

@kaibocai @cgillum

[kaibocai]

Can not reproduce from durable sidecar, I think it's a issue from durable extension. @cgillum , @davidmrdavid can you provide some guide on how to debug from durable extension on the call waitForInstanceCompletion Thank you.

[kaibocai]

The PR has been merged. Once validate will close this issue.

[kaibocai]

The root cause of the issue is the timeout comparison from https://github.com/Azure/durabletask, the issue has already been fixed at https://github.com/Azure/durabletask/pull/910.

Validated on local the fix works perfectly.

    @FunctionName("StartOrchestration")
    public HttpResponseMessage startOrchestration(
            @HttpTrigger(name = "req", methods = {HttpMethod.GET, HttpMethod.POST}, authLevel = AuthorizationLevel.ANONYMOUS) HttpRequestMessage<Optional<String>> request,
            @DurableClientInput(name = "durableContext") DurableClientContext durableContext,
            final ExecutionContext context) throws TimeoutException, InterruptedException {
        context.getLogger().info("Java HTTP trigger processed a request.");
        DurableTaskClient client = durableContext.getClient();
        String instanceId = client.scheduleNewOrchestrationInstance("Cities", "testInput1", "123");
        context.getLogger().info("Created new Java orchestration with instance ID = " + instanceId);
        HttpResponseMessage checkStatusResponse = durableContext.createCheckStatusResponse(request, instanceId);
        OrchestrationMetadata orchestrationMetadata = client.waitForInstanceCompletion(instanceId, null, true);
        System.out.println(orchestrationMetadata);
        return checkStatusResponse;
    }

Will work with @davidmrdavid on the release plan for the fix.

[kaibocai]

@kushals14, the fix is released in Durable Extension 2.9.6, please customized your DF extension to 2.9.6 to try it out. You can do that follow the steps here https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-extension-upgrade#manually-upgrade-the-durable-functions-extension.

Closing the issue.

[kushals14]

@kaibocai This app is deployed in azure and the extension is managed by extensionBundle we configured in host.json - Since I already have the manual polling in place to get the instance status, I would rather wait for this to be included in latest extension bundle.

Do you know what the timeline looks like for this to be included in latest extension bundle?

[kaibocai]

We are having some issues with the release of the extension bundle, but it should be available in the next few weeks. @lilyjma and @amamounelsayed should be able to provide more detail and a timeline for the extension bundle release.

[kushals14]

I see Durable extension 2.9.6 is available with latest extension pack - I did a test call today and it seems to be working fine for both waitForInstanceCompletion and waitForInstanceStart.