Feign limiting concurrency to 2 calls x host

jpuigsegur commented 7 years ago

We are using Feign & Hystrix implementing a service that calls other services both using @FeignClient with and without url property. I..e some of our dependencies are accessed without client load balancing (fixed url) while other are accessed using ribbon and client-side load balancing (using spring.cloud.consul). The versions we are using are:

spring boot 1.4.1.RELEASE
spring cloud feign 1.2.2.RELEASE
io.github.openfeign.feign-httpclient 9.4.0 version overrided due to a bug in version 9.3.1 (https://github.com/spring-cloud/spring-cloud-netflix/issues/1649)

One of these dependencies (ribbon load balanced) is being used with a high tps count and we are experiencing contingency consistent with a limit of aprox. 2 concurrent calls x host.

We are pretty sure that the concurrency limit is on the call itself. Looking at hystrix metrics, transaction times increase consistenly with the 2 calls x host limit. Therefore the concurrency limit must be inside Feign Hystrix command execution.
It looks that since Apache Http is on the classpath it is being used as the httpclient both in ribbon load balanced calls and url based calls.
It seems that ApacheHttp internally uses a pool that limits connections per host to 2 concurrent connections. Apache's HttpClientBuilder allows to configure this value: http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html#setMaxConnPerRoute(int) Could it be that we are hitting this limit?
While there is a way to configure other HttpClient's parameters, we haven't found any way to configure this parameter. Currently we are configuring specific timeouts with this bean:
```
@Bean
public Request.Options options() {
    return new Request.Options(connectTimeoutMillis, readTimeoutMillis);
}
```
Any help will be highly appreciated.

ryanjbaxter commented 7 years ago

Yes it is most likely you are hitting the Apache HTTP Client connection pool limit.

Http Client Creation With Ribbon: https://github.com/spring-cloud/spring-cloud-netflix/blob/master/spring-cloud-netflix-core/src/main/java/org/springframework/cloud/netflix/feign/ribbon/FeignRibbonClientAutoConfiguration.java#L92

Http Client Creation Without Ribbon: https://github.com/spring-cloud/spring-cloud-netflix/blob/master/spring-cloud-netflix-core/src/main/java/org/springframework/cloud/netflix/feign/FeignAutoConfiguration.java#L95

In both cases we are creating an ApacheHttpClient when no HttpClient bean is present. When this happens Feign just creates a basic http client https://github.com/OpenFeign/feign/blob/master/httpclient/src/main/java/feign/httpclient/ApacheHttpClient.java#L72

And its default connection pool per route is 2 https://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html

So one way around this is provide your own HttpClient bean.

FYI I am currently working on some changes that will provide us with some consistency with the creation and usage of http clients throughout Spring Cloud Netflix. You can follow https://github.com/spring-cloud/spring-cloud-netflix/issues/2026 to keep an eye on those changes.

jpuigsegur commented 7 years ago

Hi Ryan,

Thanks for your answer.

As I understood I need to provide an instance of Apache's HttpClient to Feigns context. I've tried the following:

import org.apache.http.client.HttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.springframework.context.annotation.Bean;

public class CommonValuationServiceClientConfiguration {
    @Bean
    public HttpClient httpClient() {
        return HttpClientBuilder.create()
                .setMaxConnPerRoute(200)
                .setMaxConnTotal(50)
                .build();
    }
}

This class is outside @componentScan scope and it is referenced in the corresponding Feign Client:

@FeignClient(name = "common-valuation-service",
        fallback = CommonValuationServiceClientFallbacks.class,
        configuration = CommonValuationServiceClientConfiguration.class)
public interface CommonValuationServiceClient {
[...]

However, when I run the service with this setup it seems ribbon is not activated and Feign tries to use the name as the host name:

ERROR|2017-06-22T10:41:50,302|bcef5d68237f2bf5|http-nio-8081-exec-2|GlobalExceptionHandler|CommonValuationServiceClient#valuateBookingFile(String,BookingFileDTO) failed and fallback disabled. / common-valuation-service executing POST http://common-valuation-service/common-valuation-service/1.0/valuateBookingFile/valuateBookingFile?languageId=ENG
ERROR|2017-06-22T10:41:50,302||http-nio-8081-exec-2|[dispatcherServlet]|Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.netflix.hystrix.exception.HystrixRuntimeException: CommonValuationServiceClient#valuateBookingFile(String,BookingFileDTO) failed and fallback disabled.] with root cause
java.net.UnknownHostException: common-valuation-service
        at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_131]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_131]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_131]
        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.2.jar:4.5.2]
        at feign.httpclient.ApacheHttpClient.execute(ApacheHttpClient.java:87) ~[feign-httpclient-9.4.0.jar:?]
        at org.springframework.cloud.sleuth.instrument.web.client.feign.TraceFeignClient.execute(TraceFeignClient.java:79) ~[spring-cloud-sleuth-core-1.0.10.RELEASE.jar:1.0.10.RELEASE]
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97) ~[feign-core-9.3.1.jar:?]
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76) ~[feign-core-9.3.1.jar:?]
        at feign.hystrix.HystrixInvocationHandler$1.run(HystrixInvocationHandler.java:108) ~[feign-hystrix-9.3.1.jar:?]
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:301) ~[hystrix-core-1.5.6.jar:1.5.6]
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:297) ~[hystrix-core-1.5.6.jar:1.5.6]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
        at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) ~[rxjava-1.1.10.jar:1.1.10]
        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) ~[hystrix-core-1.5.6.jar:1.5.6]
        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) ~[hystrix-core-1.5.6.jar:1.5.6]
        at org.springframework.cloud.sleuth.instrument.hystrix.SleuthHystrixConcurrencyStrategy$HystrixTraceCallable.call(SleuthHystrixConcurrencyStrategy.java:154) ~[spring-cloud-sleuth-core-1.0.10.RELEASE.jar:1.0.10.RELEASE
]
        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) ~[hystrix-core-1.5.6.jar:1.5.6]
        at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ~[rxjava-1.1.10.jar:1.1.10]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_131]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Any idea why this is happening? Without the configuration class it is working properly just with the concurrency problem.

ryanjbaxter commented 7 years ago

I believe the HTTP client bean needs to be outside of the Feign client configuration. The HTTP client bean will be used for all Feign Clients. Can you give that a try?

jpuigsegur commented 7 years ago

Thanks. It works.

You need to add this @Configuration class creates the httpClient that is used by all Feign Clients and place it in @ComponentScan path:

import org.apache.http.client.HttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ApacheHttpClientConfiguration {
    @Bean
    public HttpClient httpClient() {
        return HttpClientBuilder.create()
                .setMaxConnPerRoute(100)
                .setMaxConnTotal(1000)
                .build();
    }
}

I'll keep an eye on https://github.com/spring-cloud/spring-cloud-netflix/issues/2026

spring-cloud / spring-cloud-netflix

Feign limiting concurrency to 2 calls x host #2066