search

nfultz / grpc

gRPC clients and servers in R
72 stars 24 forks source link

Grpc Server crash #35

Open Trisfald opened 4 years ago

Trisfald commented 4 years ago

Hello, I'm trying to include this library in a suit of grpc benchmarks.

I have an issue with the server, when I send requests from a ghz client it fails with this error:

E0805 16:39:28.639090500       1 byte_buffer.cc:91]          Should never reach here.

 *** caught segfault ***
address (nil), cause 'unknown'

Traceback:
 1: run(server_functions, channel, hooks)
 2: start_server(impl, "0.0.0.0:50051")
An irrecoverable exception occurred. R is aborting now ...

It's possible to reproduce this on a linux or macos machine by just cloning this branch: https://github.com/LesnyRumcajs/grpc_bench/tree/r-bench

(the relevant code is inside the r_grpc directory)

Build the docker image with the server docker build --force-rm -f r_grpc/Dockerfile -t r_grpc_test .

Start the server docker run --rm -it --network host r_grpc_test

Start the client docker run --rm --network=host -v "${PWD}"/proto:/proto:ro --entrypoint=ghz infoblox/ghz:0.0.1 --proto=/proto/helloworld/helloworld.proto --call=helloworld.Greeter.SayHello --insecure --connections=5 --duration 5s -d "{\"name\":\"it's not as performant as we expected\"}" 127.0.0.1:50051

output of the client:

Status code distribution:
  [OK]            4 responses        
  [Unavailable]   273676 responses   
  [Canceled]      41 responses

I may be doing some silly mistake in the server implementation.

nfultz commented 4 years ago

Thank you for reporting this issue.

It's interesting that you get a few OKs and then a lot of Unavailables.

Here are my two guesses about what it could be:

  1. library versions

This package was originally built against grpc 1.3/1.4, and hadn't really been updated since. We now have a GSoC student who is updating it to use the Ubuntu 20.04 versions of libgrpc, but I'll also check that it works against the current versions in Debian testing.

  1. Server in bad state

The original release of this R package has a server that will terminate the connection after firing the R callback for a (non-streaming) service. If, in your benchmark client, you connect and make sequential calls without reconnecting in between, it probably wouldn't work yet.

Also, if you find anything in your benchmarking, we'd be interested in the results.

Trisfald commented 4 years ago

Thanks for the answer.

Currently the benchmark client is opening 5 connections and in each connection it performs 10 requests in parallel. Can this be the problem?

nfultz commented 4 years ago

It might be. If you try it with 50 connections and 1 req per connection, you might see a different distribution of success / failures.

On Wed, Aug 5, 2020 at 12:41 PM Andrea notifications@github.com wrote:

Thanks for the answer.

Currently the benchmark client is opening 5 connections and in each connection it performs 10 requests in parallel. Can this be the problem?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nfultz/grpc/issues/35#issuecomment-669448297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADGGTUYWKGTEDTPOFQXS5TR7GYVBANCNFSM4PVWFHRQ .