search

dapr / dotnet-sdk

Dapr SDK for .NET
Apache License 2.0
1.12k stars 340 forks source link

Enable specifying Grpc options like " MaxReceiveMessageSize" in WorkflowServiceCollectionExtensions #1163

Closed hsinghdeol-wpay closed 4 weeks ago

hsinghdeol-wpay commented 1 year ago

Currently dapr workflows fail with following error 2023-10-11T06:26:20.075785954Z Grpc.Core.RpcException: Status(StatusCode="ResourceExhausted", Detail="Received message exceeds the maximum configured message size.") 2023-10-11T06:26:20.075793354Z at Grpc.Net.Client.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken) 2023-10-11T06:26:20.075801654Z at Grpc.Net.Client.Internal.HttpContentClientStreamReader2.MoveNextCore(CancellationToken cancellationToken) 2023-10-11T06:26:20.075833955Z at Microsoft.DurableTask.GrpcExtensions.ReadAllAsync[T](IAsyncStreamReader1 reader, CancellationToken cancellationToken)+MoveNext() 2023-10-11T06:26:20.075841955Z at Microsoft.DurableTask.GrpcExtensions.ReadAllAsync[T](IAsyncStreamReader1 reader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() 2023-10-11T06:26:20.075852855Z at Microsoft.DurableTask.Worker.Grpc.GrpcDurableTaskWorker.Processor.ProcessWorkItemsAsync(AsyncServerStreamingCall1 stream, CancellationToken cancellation) 2023-10-11T06:26:20.075860455Z at Microsoft.DurableTask.Worker.Grpc.GrpcDurableTaskWorker.Processor.ProcessWorkItemsAsync(AsyncServerStreamingCall1 stream, CancellationToken cancellation) 2023-10-11T06:26:20.075866755Z at Microsoft.DurableTask.Worker.Grpc.GrpcDurableTaskWorker.Processor.ExecuteAsync(String target, CancellationToken cancellation) I found out that there is no way to override default limit of 4mb in the following file. https://github.com/dapr/dotnet-sdk/blob/3b979e6bdb1d779563f1656fa684183b2bfecd08/src/Dapr.Workflow/WorkflowServiceCollectionExtensions.cs#L159

There should be way to override default GRPC options

zminot commented 10 months ago

I've run into this as well. I know this was referenced in https://github.com/dapr/dotnet-sdk/issues/978#issuecomment-1760474157, but also bringing this up in this issue.

It seems like DurableTasks has a hard limit of 4MB as noted in https://github.com/microsoft/durabletask-dotnet/issues/150#issuecomment-1569165595, and the Dotnet SDK for Dapr Workflow uses that underneath. Do we think it'd be a good idea to update the Dapr documentation for this if this is now a known (and presumedly unsolvable by Dapr) issue?

Not sure if https://github.com/microsoft/durabletask-dotnet/issues/154 is something that will alleviate this, however.

cgillum commented 2 months ago

I don't think this should require any work on the Durable Task SDK side since the Dapr Workflow client already controls its gRPC channel configuration as referenced in the OP. I can imagine that we may need to expose this in the sidecar as well, but it's possible this may already be supported if it's a server setting (durabletask-go is part of the same server as the rest of the Dapr APIs).

Agreed that this is important to fix given that it can cause workflows to fail mid-execution.

WhitWaldo commented 4 weeks ago

Should be resolved by #1244