Increase 4MB of gRPC limited in Azure Durable Function for Java

[kevin808]

There seems to be a limit only in Azure Durable Function For Java that cannot send payload over 4MB, can you update the limit and align with int.MaxValue in function host runtime? Below are the observation about this issue:

1. Got below error when POST to durable function with a payload over 4MB.

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.StartOrchestration ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure

Exception: StatusRuntimeException: RESOURCE_EXHAUSTED: Received message larger than max (5060857 vs. 4194304) Stack: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22) at com.microsoft.azure.functions.worker.broker.EnhancedJavaMethodExecutorImpl.execute(EnhancedJavaMethodExecutorImpl.java:22) at com.microsoft.azure.functions.worker.chain.FunctionExecutionMiddleware.invoke(FunctionExecutionMiddleware.java:19) at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21) at com.microsoft.durabletask.azurefunctions.internal.middleware.OrchestrationMiddleware.invoke(OrchestrationMiddleware.java:29) at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21) at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:125) at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:34) at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10) at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:44) at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:94) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Received message larger than max (5060857 vs. 4194304) at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) at com.microsoft.durabletask.implementation.protobuf.TaskHubSidecarServiceGrpc$TaskHubSidecarServiceBlockingStub.startInstance(TaskHubSidecarServiceGrpc.java:919) at com.microsoft.durabletask.DurableTaskGrpcClient.scheduleNewOrchestrationInstance(DurableTaskGrpcClient.java:114) at com.microsoft.durabletask.DurableTaskClient.scheduleNewOrchestrationInstance(DurableTaskClient.java:67) at com.microsoft.durabletask.DurableTaskClient.scheduleNewOrchestrationInstance(DurableTaskClient.java:52) at com.singpost.wms.uniqlo.function.SplitXMLFunction.startOrchestration(SplitXMLFunction.java:75) ... 20 more

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at //src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs : 96 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at //src/WebJobs.Script/Description/FunctionInvokerBase.cs : 82 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task1 src) at /_/src/WebJobs.Script/Description/FunctionGenerator.cs : 225 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 52 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 581 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 527 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 306 End of inner exception at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 352 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 108

2. It is regarding the grpcMaxMessageLength, there are multiple issues related to it. In Function Host Runtime, the limit should be set to int.MaxValue:

https://github.com/Azure/azure-functions-host/issues/2677

DefaultMaxMessageLengthBytes = int.MaxValue; https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/Rpc/RpcWorkerConstants.cs#L38

3. From logs, we see that grpcMaxMessageLength has already been set to 2147483647. Howeve it does not seem to take place in the durable function.

Starting worker process with FileName:C:\Program Files\Java\microsoft-jdk-11.0.16.1/bin/java WorkingDirectory:C:\home\site\wwwroot Arguments: -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -noverify -Djava.net.preferIPv4Stack=true -jar "C:\Program Files (x86)\SiteExtensions\Functions\4.19.0\workers\java\azure-functions-java-worker.jar" --host 127.0.0.1 --port 50628 --workerId ada93a63-8e10-49ac-ba64-19113424c536 --requestId d5533bd6-b6a2-442d-b394-22c67ae1f6b5 --grpcMaxMessageLength 2147483647

4. This issue can always be reproduced from local with latest function host version: 4.21.1.20667 when send a POST with payload larger than 4MB.

5. This issue cannot be reproduced when use Azure Durable Function for Python. Also, no issues with HTTP trigger in normal Java function app. Hence we believe it is only with Azure Durable Function for Java.

6. There is no such setting for "grpc max message size" in this repo found, it seems to inherit default 4MB limit in gRPC.

[cgillum]

This seems to be the same issue as https://github.com/microsoft/durabletask-dotnet/issues/150.

[jviau]

Work is happening to add official gRPC extension support to the functions host and workers. When this happens, we will inherit whatever gRPC max message limit the host has configured is. Only work needed here will be to eventually switch to the gRPC connection provided by the functions java extension packages when it is available.