Poor gRPC performance on test

alexyakunin commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug

I wrote a small benchmark for some of mainstream RPC libraries on .NET - originally to benchmark my own library (Stl.Rpc), but ended up adding nearly every other option.

Benchmark description and its results:

https://servicetitan.github.io/Stl.Fusion.Samples/rpc-benchmark
Raw version of the same document: https://github.com/servicetitan/Stl.Fusion.Samples/blob/master/docs/rpc-benchmark.md

Expected Behavior

gRPC is expected to perform on SignalR level. Instead, it performs ~ on HttpClient level.

To clarify, the difference between gRPC and SignalR performance is dramatic:

101.81K calls/s vs 345.35K calls/s on SayHello LAN test - it's the same payload as in https://github.com/LesnyRumcajs/grpc_bench - and other LAN tests with smaller payloads perform even worse
125.11K calls/s vs 1.22M calls/s on the same SayHello test running server + client on a single machine.

Steps To Reproduce

Clone the repository: git clone git@github.com:servicetitan/Stl.Fusion.Samples.git
Run dotnet run -c Release --project src/RpcBenchmark/RpcBenchmark.csproj

See https://servicetitan.github.io/Stl.Fusion.Samples/rpc-benchmark to try some other run options.

Exceptions (if any)

No response

.NET Version

8.0 Preview 7

Anything else?

No response

ShreyasJejurkar commented 1 year ago

cc @JamesNK!

JamesNK commented 1 year ago

SignalR client creates an HTTP request when it connects, then streams data over the HTTP request when making calls. gRPC creates a new HTTP request for each call. It's more expensive for a client and server to process a new HTTP request.

A more equivalent comparison would be the performance of SignalR vs gRPC bidirectional streaming.

I don't have time to do a performance investigation of every benchmark. The benchmarks in the grpc-dotnet and grpc_bench repos are a good comparison if you want to see if performance is lower than expected.

alexyakunin commented 1 year ago

Hi @JamesNK , thanks for the response - esp. taking into account your time constraints. I asked this question more to understand whether it's normal at all to see that kind of a difference between SignalR calls and gRPC calls - coz honestly, I was a bit shocked at first.

If you feel that it's normal (esp. from your experience with some other benchmarks), I don't think there is anything to investigate. I also asked a couple other ppl who are familiar with gRPC much more than I am, so right now I feel more confident this is not a mistake in settings or something similar on my end.

As for:

grpc_bench: all of my benchmarking code is effectively based on theirs - i.e. my "SayHello" is almost 100% identical to the code from this benchmark; it also uses the same payload they use on perf tests - I wanted the results from my "SayHello" to be directly comparable with theirs
grpc-dotnet/perf - it feels it does quite a bit more than "SayHello" and tests a set of advanced features (compression, headers, etc.), so I am not quite sure if it makes sense to dig deeper there.

What would be great if you could point on some repository that shows a perfect client & server setup tuned for max gRPC throughput.

alexyakunin commented 1 year ago

Also,

A more equivalent comparison would be the performance of SignalR vs gRPC bidirectional streaming.

I understand that bi-di streaming would offer a higher throughput, but I think it's not quite right to use streaming on RPC test. I.e. if there would be a built-in wrapper that routes RPC/RPC-like calls to gRPC streaming layer, I would be happy to add it to this test, but otherwise it's not apples-to-apples comparison.

I plan to add streaming tests in future though, so if you're interested, I would be happy to let you know once I have some results.

dotnet / aspnetcore