search

PlayFab / consuldotnet

.NET API for Consul (http://www.consul.io/)
Apache License 2.0
692 stars 193 forks source link

Consul.ConsulRequestException: Unexpected response, status code InternalServerError: rpc error making call: EOF #110

Closed vasily-kirichenko closed 6 years ago

vasily-kirichenko commented 6 years ago

https://github.com/Horusiath/Akka.Cluster.Discovery uses this client, see https://github.com/Horusiath/Akka.Cluster.Discovery/blob/ca04b544568fe700355d30a131b8a8362e7ae1fc/src/Akka.Cluster.Discovery.Consul/ConsulDiscoveryService.cs#L34. After a week of running without problems, I got the following exception:

03.02.2018 02:51:28.308 ERROR [Akka.Cluster.Discovery.Consul.ConsulDiscoveryService,  944] - Couldn't retrieve collection of nodes from external service. Is current node unreachable? Shutting down... Consul.ConsulRequestException: Unexpected response, status code InternalServerError: rpc error making call: EOF
   at Consul.GetRequest`1.<Execute>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Akka.Cluster.Discovery.Consul.ConsulDiscoveryService.<GetNodesAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Akka.Cluster.Discovery.DiscoveryService.<ReconcileClusterStateAsync>d__19.MoveNext()

(it's thrown here https://github.com/Horusiath/Akka.Cluster.Discovery/blob/ca04b544568fe700355d30a131b8a8362e7ae1fc/src/Akka.Cluster.Discovery.Consul/ConsulDiscoveryService.cs#L57)

Is it idiomatic to use ConsulClient this way (I mean create it once and use forever)? If it's not, is creating an instance for each request a nice idea? I see all the state is lazy, so I suspect it's quite expensive to recreate every time?

highlyunavailable commented 6 years ago

Yes, it is idiomatic to create it once per distinct cluster/datacenter connection and then reuse it forever. It's mostly that way due to the fact that HttpClient should be used in that way as well.

vasily-kirichenko commented 6 years ago

@highlyunavailable I'm not sure I understand you. I do reuse the client and it dies.

highlyunavailable commented 6 years ago

Re-make the request then? Or is the problem that you can no longer issue any requests?

highlyunavailable commented 6 years ago

The error Unexpected response, status code InternalServerError: rpc error making call: EOF means that Consul itself is encountering an error - the words "rpc error making call: EOF" are from Go.