Connection Pool Always with 500 Connections

Hi,

We started to see performance problems in our consul infrastructure because we observed all our java microservices are using the full pool of connections. We are using sping boot cloud consul library that uses your "consul-api" library as an interface with consul.

Running netstat in consul host, we can observe all of the servers are in the maximum number of connections allowed by this library:

netstat -anlp |grep 8500  | awk '{ print $5}' |cut -f "1" -d ":" | sort | uniq -c
    501 172.17.0.10
    500 172.17.0.12
    500 172.17.0.13
    501 172.17.0.14
    500 172.17.0.15
    500 172.17.0.17
    501 172.17.0.18
    500 172.17.0.19
    500 172.17.0.2
    500 172.17.0.25
    501 172.17.0.26
    500 172.17.0.28
    500 172.17.0.3
    501 172.17.0.5
    501 172.17.0.6
    500 172.17.0.7
    500 172.17.0.8
    500 172.17.0.9

Spring-cloud-consul have an issue closed that seems related to this issue https://github.com/spring-cloud/spring-cloud-consul/issues/549 After a further reading around the Internet, we found this issue in httpclient library https://issues.apache.org/jira/browse/HTTPCLIENT-2007

Seems that httpclient library is not reusing the https connections, so we implemented:

                HttpClientBuilder httpClientBuilder = HttpClientBuilder.create().
                    setConnectionManager(connectionManager).
                    setDefaultRequestConfig(requestConfig).
                                        setUserTokenHandler(new NoopUserTokenHandler());

With this patch, the connections used by spring-boot-consul dropped down from 500 to 2 after 30 minutes of testing.

We are not experts with the httpclient library but seems there's some problem with the implementation of the pool.

Ecwid / consul-api

Connection Pool Always with 500 Connections #235