Performance problem when using the client in multiple threads at the same time

[brunczelandras]

I am using GrpcWebHandler to make my .Net Framework client working. I am using Win10. Throughput of the client requests is decreasing significantly when the client sends the requests in multiple threads at the same time. After some investigation I see that sporadically there is around 1 second timeframe when there are no requests sent or responses received at all. Rest of the time the requests are processed in few milliseconds. If the client sends e.g. 10 requests in parallel then it happens 1 or 2 times. If the client sends the requests sequentially then everything is fine. I am using version 2.51.0.

Proto file

syntax = "proto3";

message Request
{
    int32 value = 1;
}
message Response
{
    int32 value = 1;
}
service MyService
{
    rpc DoWork (Request) returns (Response);
}

Client (.Net Framework)

using Grpc.Net.Client;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading.Tasks;

namespace MyTest
{
    public class Program : IDisposable
    {
        private static readonly Program myProgram = new Program();
        private MyService.MyServiceClient myClient;
        private readonly GrpcChannel myChannel;
        private readonly object myChannelLock = new object();
        private static int myNextRequestId;

        public static void Main(string[] args)
        {
            var sw = new Stopwatch();
            sw.Start();
            int numberOfClients = int.Parse(args[0]);
            var taskList = new List<Task>();

            for (int i = 0; i < numberOfClients; i++)
            {
                taskList.Add(Task.Run(Test)); // this is slow
                //Test(); // this is fast
            }

            Parallel.ForEach(taskList, t => t.Wait());
            sw.Stop();

            Console.WriteLine("total {0} ms", sw.Elapsed.TotalMilliseconds);
        }

        private static void Test()
        {
                var sw = new Stopwatch();
                sw.Start();
                myProgram.DoWork();
                sw.Stop();
                Console.WriteLine("DoWork {0} ms", sw.Elapsed.TotalMilliseconds);
        }

        public void DoWork()
        {
            var request = new Request { Value = myNextRequestId++ };
            var response = myClient.DoWork(request);
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected virtual void Dispose(bool disposing)
        {
            if (disposing)
            {
                lock (myChannelLock)
                {
                    if (myChannel != null)
                    {
                        myChannel.ShutdownAsync().Wait(1000);
                        myChannel.Dispose();
                    }
                    myClient = null;
                }
            }
        }

        public Program()
        {
            lock (myChannelLock)
            {
                myChannel = GrpcClientFactory.CreateChannel();
                myClient = GrpcClientFactory.CreateClient<MyService.MyServiceClient>(myChannel);
            }
        }
    }
}

GrpcClientFactory.cs

using System;
using System.Net.Http;
using Grpc.Net.Client;
using Grpc.Net.Client.Web;

namespace MyTest
{ 
    internal static class GrpcClientFactory
    {
        internal static T CreateClient<T>()
        {
            return CreateClient<T>(CreateChannel());
        }

        internal static T CreateClient<T>(GrpcChannel channel)
        {
            return (T)Activator.CreateInstance(typeof(T), channel);
        }

        internal static GrpcChannel CreateChannel()
        {
                var channel = GrpcChannel.ForAddress($"http://localhost:47102", new GrpcChannelOptions
                {
                    HttpHandler = new GrpcWebHandler(new HttpClientHandler())
                    {
                        HttpVersion = new Version(1, 1)
                    }
                });
                return channel;
        }
    }
}

Service (.Net 6)

using System.IO;
using System.Net;
using System.Reflection;
using System.Threading.Tasks;

using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

namespace MyServer
{
    class Program
    {
        static void Main()
        {
            var hostBuilder = CreateHostBuilder();
            var host = hostBuilder.Build();
            var task = Task.Run(() => host.Run());
            task.Wait();
        }

        public static IHostBuilder CreateHostBuilder()
        {
            return CreateHostBuilderBase()
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.ConfigureKestrel((_, options) => options.Listen(IPAddress.Any, 47102, listenOptions =>
                    {
                        listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
                    })); webBuilder.UseStartup<MyServiceStartup>();
                });
        }

        internal static IHostBuilder CreateHostBuilderBase()
        {
            return Host.CreateDefaultBuilder()
                .ConfigureAppConfiguration(app =>
                {
                    app.SetBasePath(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location));
                    app.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true);
                })
                .ConfigureLogging(logging =>
                {
                    logging.ClearProviders();
                    logging.AddDebug();
                    logging.AddSimpleConsole(options =>
                    {
                        options.IncludeScopes = true;
                        options.SingleLine = true;
                        options.TimestampFormat = "yyyy-MM-dd HH:mm:ss.fff ";
                    });
                });
        }
    }
}

MyServiceStartup.cs

using System;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace MyServer
{
    internal class MyServiceStartup
    {
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddGrpc();
            services.AddSingleton<MyServiceImpl>();
            services.AddCors();
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseRouting();
            app.UseGrpcWeb();

            app.UseCors(x => x
                .AllowAnyOrigin()
                .AllowAnyMethod()
                .AllowAnyHeader());

            app.UseHttpsRedirection();

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapGrpcService<MyServiceImpl>().EnableGrpcWeb();
            });

            app.ApplicationServices.GetRequiredService<MyServiceImpl>();

            Console.WriteLine("Service started.");
        }
    }
}

MyServiceImpl.cs

using System.Threading.Tasks;
using Grpc.Core;
using Microsoft.Extensions.Logging;

namespace MyServer
{
    internal sealed class MyServiceImpl : MyService.MyServiceBase
    {
        public MyServiceImpl(ILoggerFactory loggerFactory)
        {
            myLogger = loggerFactory.CreateLogger<MyServiceImpl>();
        }

        public override Task<Response> DoWork(Request request, ServerCallContext context)
        {
            return Task.Run(() =>
            {
                myLogger.LogInformation("DoWork started");
                return new Response { Value = request.Value };
            });
        }

        private readonly ILogger<MyServiceImpl> myLogger;
    }
}

Output

DoWork 1993,2582 ms
DoWork 1993,1801 ms
DoWork 1993,8685 ms
DoWork 1993,2391 ms
DoWork 1993,1821 ms
DoWork 1993,1562 ms
DoWork 1993,1521 ms
DoWork 1993,2175 ms
DoWork 1993,2744 ms
DoWork 1993,2378 ms
total 2008,372 ms

[AloisKraus]

Is this a known limitation or what is the state of this issue? Are concurrent requests by GRPC not supported?

[sampajano]

Hi :)

Hi! Thanks for reporting this issue!

Although, .NET server is not developed as part of this project.

Maybe https://github.com/grpc/grpc-dotnet would be a more appropriate place for this report?

[brunczelandras]

@sampajano Hi, There's no performance degradation if I use grpc client without GrpcWebHandler so I don't think that the grpc-dotnet project should handle this issue.

[sampajano]

@brunczelandras Ah. Understand your concern..

Although, the server implementations / performance are outside of the concern of this repo, as this repo hosts only the Javascript client.

The official solution we support is Envoy + gRPC server, and hence .NET is out of the scope.

I think you would have better luck reporting the issue where the server code is developed. :)

[sampajano]

Closing for now. Feel free to reopen if you still have questions :)