search

ContainX / openstack4j

A Fluent OpenStack SDK / Client Library for Java
http://openstack4j.com
Other
289 stars 368 forks source link

Problem in connection pooling using httpclient as connector #579

Open sjpatil opened 8 years ago

sjpatil commented 8 years ago

We are using 2.10 release with httpclient and facing issue with connection pool.

We are doing multiple server snapshot downloads in a sequence. First download works well but later downloads get stuck. As by default the max connections per route is 2; when increased withMaxConnectionsPerRoute , we could do more successful calls. But ultimately it gets stuck again after the connections are exhausted. It seems connections are open and not getting closed.

Config config=Config.newConfig().withMaxConnectionsPerRoute(4)

Logs where it gets stuck are -

2016/02/23 08:47:41:573 UTC [DEBUG] PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://endpoint:5000] can be kept alive indefinitely 2016/02/23 08:47:41:573 UTC [DEBUG] PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://endpoint:5000][total kept alive: 2; route allocated: 1 of 2; total allocated: 4 of 20] 2016/02/23 08:47:41:574 UTC [DEBUG] RequestAddCookies - CookieSpec selected: best-match 2016/02/23 08:47:41:574 UTC [DEBUG] RequestAuthCache - Auth cache not set in the context 2016/02/23 08:47:41:574 UTC [DEBUG] PoolingHttpClientConnectionManager - Connection request: [route: {}->http://endpoint:8774][total kept alive: 2; route allocated: 2 of 2; total allocated: 4 of 20]

Our download code makes sure that the input stream that we receive is closed. However, all operations from ServerService get stuck after the last successful download, however other calls from ImageService can be made.

vinodborole commented 8 years ago

@gondor Can you please help us here?

sachin-walia commented 8 years ago

@vinodborole is this still an issue?

dbantchovski commented 8 years ago

@vinodborole

Yes, this is still an issue. I am facing a similar problem /http connection leak when using apache http client connector/. My scenario is complex, but I managed to extract the problematic part from it:

Server server = os.compute().servers().get(serverId); os.compute().servers().createSnapshot(serverId, "one"); os.compute().servers().createSnapshot(serverId, "two"); os.compute().servers().createSnapshot(serverId, "three");

The third invocation hangs, waiting to obtain a connection - /not that by default apache http client uses pool of 20, connections, but only 2 per host/:

`java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(ZJ)V(Native Method)

Looking at the code I didn't find response.close(), and this could be a problem. I will check the code again and will keep you updated with my findings.

dbantchovski commented 8 years ago

I proposed a fix regarding my use-case – createSnapshot(); My initial assumption was, that these both issues are somehow related. An easy way to reproduce the initial problem:

InputStream in1 = os.images().getAsStream(imageId); InputStream in2 = os.images().getAsStream(imageId); InputStream in3 = os.images().getAsStream(imageId);

Looking at code org.openstack4j.openstack.image.internal.ImageServiceImpl. getAsStream(String imageId) I found a similar situation – the response in not consumed nor closed and this prevent connection to be returned in the pool.

Unfortunately I am not sure how to fix it, since the connection should stay open as long as the returned stream is been used.

vinodborole commented 8 years ago

Thanks @dbantchovski.

even i am not sure how to fix it; there was similar ticket opened few months back and that was marked as fixed; seems it was not validated quite well.

Currently, we have switched the connector to jersey2

olivergondza commented 7 years ago

Observing the same problem with openstack4j/openstack4j-httpclient 3.0.4 occasionally without any special triggering conditions.

Thread stuck reading the socket:

"Computer.threadPoolForRemoting [#23344]" #537821 daemon prio=5
   java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
  at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
  at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
  at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
  at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
  at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
  at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
  at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
  at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
  at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
  at org.openstack4j.connectors.httpclient.HttpCommand.execute(HttpCommand.java:112)
  at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.invokeRequest(HttpExecutorServiceImpl.java:65)
  at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.invoke(HttpExecutorServiceImpl.java:56)
  at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.execute(HttpExecutorServiceImpl.java:32)
  at org.openstack4j.core.transport.internal.HttpExecutor.execute(HttpExecutor.java:51)
  at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:208)
  at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:202)
  at org.openstack4j.openstack.compute.internal.ServerServiceImpl.boot(ServerServiceImpl.java:131)
  at org.openstack4j.openstack.compute.internal.ServerServiceImpl.bootAndWaitActive(ServerServiceImpl.java:139)
  ...

   Locked ownable synchronizers:
  - <0x5084265d> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Dozens of other threads like this blocked because of it:

"jenkins.util.Timer [#2]" #48 daemon prio=5
   java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for <0x2e9dbd87> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:138)
  at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
  at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
  at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
  at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
  at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
  at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
  at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
  at org.openstack4j.connectors.httpclient.HttpCommand.execute(HttpCommand.java:112)
  at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.invokeRequest(HttpExecutorServiceImpl.java:65)
  at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.invoke(HttpExecutorServiceImpl.java:56)
  at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.execute(HttpExecutorServiceImpl.java:32)
  at org.openstack4j.core.transport.internal.HttpExecutor.execute(HttpExecutor.java:51)
  at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:208)
  at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:202)
  at org.openstack4j.openstack.compute.internal.ServerServiceImpl.list(ServerServiceImpl.java:97)
  at org.openstack4j.openstack.compute.internal.ServerServiceImpl.list(ServerServiceImpl.java:82)
 ...

   Locked ownable synchronizers:
  - <0x8691939> (a java.util.concurrent.ThreadPoolExecutor$Worker)
  - <0x69b19783> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

Switching to okhttp connector seem to resolve the problem for now. I hope I am not too optimistic to believe the ConnectionException{message=timeout, status=0}s thrown are the same underlying problem handled better.