search

nfultz / grpc

gRPC clients and servers in R
72 stars 24 forks source link

too many open files #17

Open daroczig opened 6 years ago

daroczig commented 6 years ago

@nfultz, did you see any issues with letting the service run for a longer time?

Everything seems to work perfectly fine while running a grpc service in Docker for a while, and the service responds to requests, but after ~25 mins the process fails with:

{"log":"E0106 00:37:51.980090342       1 tcp_server_posix.c:245]     Failed accept4: Too many open files\n","stream":"stderr","time":"2018-01-06T00:37:51.98024391Z"}

Any chance that the server does not free up some resources in the loops or any other ideas?

nfultz commented 6 years ago

I think this is an upstream issue - https://github.com/grpc/grpc/issues/11015

nfultz commented 6 years ago

Are you setting up the docker like inst/docker-dev or inst/docker-stable ? You might try the other way.

daroczig commented 6 years ago

Good point! I'm using the pretty outdated debian version, will check with a newer version and let you know what I've found.

nfultz commented 6 years ago

Just not too new, current ubuntu also has a broken libgrpc :(

nfultz commented 6 years ago

Also unclear at this point if the grpc cleanup code path is getting executed correctly - do you get this error only when the service is under load or does it do error out even when it's idle?

daroczig commented 6 years ago

It's been failing after ~20-25 mins of the last request, so while it's idle -- on the other hand, I've not tried the service for a longer time period with decent load. So far I've just started a dozen of Docker containers and tested with a few queries and then decided to leave those for the weekend -- when I found out about this issue. I will keep you posted if I find something useful (eg on grpc version).

daroczig commented 6 years ago

Sorry, did not have the time to look into this in more details -- meanwhile, I'm just restarting the containers every 15 mins as a workaround.