search

grpc / grpc-dotnet

gRPC for .NET
Apache License 2.0
4.18k stars 769 forks source link

Performance Issue: Unable to Exceed 5000 RPC Requests per Second Using .NET gRPC Client #2076

Closed aaron-ai closed 1 year ago

aaron-ai commented 1 year ago

We are currently implementing a RocketMQ gRPC client using .NET gRPC, as a communication layer for RocketMQ. Our client and server have approximately 10 ms of network latency, and we are using async await to implement message sending. However, we have found that we can only send up to approximately 5000 messages (each message sending corresponds to an unary RPC request) per second before receiving an RPC timeout error. We tried to establish multiple gRPC channels to improve performance, but it didn't work. Can someone help us identify the reason for this issue?

We also used the gRPC Java client to perform performance testing and found that we did not have this issue: https://github.com/apache/rocketmq-clients/tree/master/java

Here is the benchmark program we are using: https://github.com/apache/rocketmq-clients/blob/master/csharp/examples/ProducerBenchmark.cs

Here is the code we are using for the gRPC client: https://github.com/apache/rocketmq-clients/blob/master/csharp/rocketmq-client-csharp/RpcClient.cs

Platform: .NET Core 3.1 and .NET 5+, server gc mode is enabled

JamesNK commented 1 year ago

.NET Core 3.1 and .NET 5 aren't supported. What happens on .NET 6?

What is the timeout? A very small deadline could trigger once the server gets some load and can't handle each call immediately.

Is there a simple way I can run your test to see for myself?