jkirschner-hashicorp
commented
2 years ago Hi,
It looks like you're using Consul as a service discovery backend for Ocelot. I don't have any personal experience with Ocelot, so I'll give you some information from the Consul perspective and you may need to sort out how/if it can be applied from Ocelot.
Can you tell which version of Consul you are on?
Consul agents have a config parameter called http_max_conns_per_client that:
Configures a limit of how many concurrent TCP connections a single client IP address is allowed to open to the agent's HTTP(S) server.
This limit defaults to 200 concurrent TCP connections in recent versions of Consul. You may find additional context on GH-7527 suggesting that some SDKs and applications don't necessarily close TCP connections when they are no longer needed. GH-7527 was closed by a PR that caused HTTP 429 to be returned so that SDKs and applications using Consul have an indication that they need to better manage their connections.
Is it possible to pass the http_max_conns_per_client configuration from Consul to Ocelot? Or to contact Ocelot about whether they should change how TCP connections are handled?
Consul.ConsulRequestException: Unexpected response, status code TooManyRequests: Your IP is issuing too many concurrent connections, please rate limit your calls
at Consul.GetRequest`1.Execute(CancellationToken ct) at Ocelot.Provider.Consul.Consul.Get() at Ocelot.LoadBalancer.LoadBalancers.LeastConnectionCreator.<>c__DisplayClass0_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Ocelot.LoadBalancer.LoadBalancers.LeastConnection.Lease(HttpContext httpContext)
at Ocelot.LoadBalancer.Middleware.LoadBalancingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.DownstreamPathManipulation.Middleware.ClaimsToDownstreamPathMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.QueryStrings.Middleware.ClaimsToQueryStringMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Headers.Middleware.ClaimsToHeadersMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Authorisation.Middleware.AuthorisationMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Claims.Middleware.ClaimsToClaimsMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Authentication.Middleware.AuthenticationMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.RequestId.Middleware.RequestIdMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.RateLimit.Middleware.ClientRateLimitMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Request.Middleware.DownstreamRequestInitialiserMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Headers.Middleware.HttpHeadersTransformationMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Security.Middleware.SecurityMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Multiplexer.MultiplexingMiddleware.Fire(HttpContext httpContext, RequestDelegate next)
at Ocelot.Multiplexer.MultiplexingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Responder.Middleware.ResponderMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.MiddlewareAnalysis.AnalysisMiddleware.Invoke(HttpContext httpContext)
at Ocelot.Errors.Middleware.ExceptionHandlerMiddleware.Invoke(HttpContext httpContext)