Adopt the Functions Host gRPC server

[bytemech]

Hi everyone, I'd like to raise this as an enhancement request.

Back in 2020, the gRPC message limit was raised to int.MaxValue in [pack]Increase gRpc Server max message size #5801 in the azure-functions-host runtime.

However, Durable Functions persists with it's own implementation of gRPC, even though @jviau has mentioned that the intention is to remove this implementation. See:

• https://github.com/microsoft/durabletask-java/issues/137
• https://github.com/Azure/azure-functions-durable-js/issues/93 (It appears the JS implementation has raised the limit themselves on their implementation, which I'm not sure is gRPC-related)

Currently trying to pass a large payload to an orchestration function (dotnet-isolated / 7) results in the following error:

[2023-06-11T09:43:23.451Z] Worker process started and initialized.
[2023-06-11T09:43:25.814Z] Executing 'Functions.funcDurableFunctionsClient' (Reason='This function was programmatically called via the host APIs.', Id=71a2086b-78a8-44a9-afc6-5ab31fdfd706)
[2023-06-11T09:43:27.278Z] Scheduling new funcDurableFunctionsOrchestration orchestration with instance ID '1c2eb812d890432d9f7800b4c642b8ad' and 43683417 bytes of input data.
[2023-06-11T09:43:27.401Z] Executed 'Functions.funcDurableFunctionsClient' (Failed, Id=71a2086b-78a8-44a9-afc6-5ab31fdfd706, Duration=1791ms)
[2023-06-11T09:43:27.402Z] System.Private.CoreLib: Exception while executing function: Functions.funcDurableFunctionsClient. System.Private.CoreLib: Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="ResourceExhausted", Detail="Received message larger than max (43683498 vs. 4194304)"))
[2023-06-11T09:43:27.403Z]  ---> Grpc.Core.RpcException: Status(StatusCode="ResourceExhausted", Detail="Received message larger than max (43683498 vs. 4194304)")
[2023-06-11T09:43:27.403Z]    at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.ScheduleNewOrchestrationInstanceAsync(TaskName orchestratorName, Object input, StartOrchestrationOptions options, CancellationToken cancellation)
[2023-06-11T09:43:27.404Z]    at Example.DurableFunctions.DurableFunctionsClient.funcDurableFunctionsClient(HttpRequestData req, DurableTaskClient client, FunctionContext executionContext) in C:\Users\gloverb\src\durable-functions-example\src\DurableFunctionsHTTPClient.cs:line 31
[2023-06-11T09:43:27.405Z]    --- End of inner exception stack trace ---
[2023-06-11T09:43:27.406Z]    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-06-11T09:43:27.406Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-06-11T09:43:27.407Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-06-11T09:43:27.408Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-06-11T09:43:27.409Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-06-11T09:43:27.410Z] --- End of stack trace from previous location ---
[2023-06-11T09:43:27.410Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-06-11T09:43:27.411Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-06-11T09:43:27.412Z] --- End of stack trace from previous location ---
[2023-06-11T09:43:27.412Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
[2023-06-11T09:43:27.413Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-06-11T09:43:27.414Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-06-11T09:43:27.415Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82
Stack:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2023-06-11T09:43:27.416Z]    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2023-06-11T09:43:27.417Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2023-06-11T09:43:27.417Z]    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2023-06-11T09:43:27.418Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-06-11T09:43:27.419Z] --- End of stack trace from previous location ---
[2023-06-11T09:43:27.419Z]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2023-06-11T09:43:27.420Z]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2023-06-11T09:43:27.421Z] --- End of stack trace from previous location ---
[2023-06-11T09:43:27.421Z]    at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
[2023-06-11T09:43:27.422Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2023-06-11T09:43:27.423Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 22
[2023-06-11T09:43:27.424Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82.
[2023-06-11T09:43:28.033Z] Host lock lease acquired by instance ID '0000000000000000000000002AF9CF2F'.

[alievense]

We're currently facing the same problem when reading a 5MB output of an instance.

var instance = await client.GetInstancesAsync(instanceId, true);

[krishna-vijas]

Is there any workaround for this as of now?

[kemmis]

+1 to fix this. I'm getting ResourceExhausted when sending payloads > 4 MB.