Open sjpatil opened 8 years ago
@gondor Can you please help us here?
@vinodborole is this still an issue?
@vinodborole
Yes, this is still an issue. I am facing a similar problem /http connection leak when using apache http client connector/. My scenario is complex, but I managed to extract the problematic part from it:
Server server = os.compute().servers().get(serverId); os.compute().servers().createSnapshot(serverId, "one"); os.compute().servers().createSnapshot(serverId, "two"); os.compute().servers().createSnapshot(serverId, "three");
The third invocation hangs, waiting to obtain a connection - /not that by default apache http client uses pool of 20, connections, but only 2 per host/:
`java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(ZJ)V(Native Method)
Looking at the code I didn't find response.close(), and this could be a problem. I will check the code again and will keep you updated with my findings.
I proposed a fix regarding my use-case – createSnapshot(); My initial assumption was, that these both issues are somehow related. An easy way to reproduce the initial problem:
InputStream in1 = os.images().getAsStream(imageId); InputStream in2 = os.images().getAsStream(imageId); InputStream in3 = os.images().getAsStream(imageId);
Looking at code org.openstack4j.openstack.image.internal.ImageServiceImpl. getAsStream(String imageId) I found a similar situation – the response in not consumed nor closed and this prevent connection to be returned in the pool.
Unfortunately I am not sure how to fix it, since the connection should stay open as long as the returned stream is been used.
Thanks @dbantchovski.
even i am not sure how to fix it; there was similar ticket opened few months back and that was marked as fixed; seems it was not validated quite well.
Currently, we have switched the connector to jersey2
Observing the same problem with openstack4j
/openstack4j-httpclient
3.0.4 occasionally without any special triggering conditions.
Thread stuck reading the socket:
"Computer.threadPoolForRemoting [#23344]" #537821 daemon prio=5
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.openstack4j.connectors.httpclient.HttpCommand.execute(HttpCommand.java:112)
at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.invokeRequest(HttpExecutorServiceImpl.java:65)
at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.invoke(HttpExecutorServiceImpl.java:56)
at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.execute(HttpExecutorServiceImpl.java:32)
at org.openstack4j.core.transport.internal.HttpExecutor.execute(HttpExecutor.java:51)
at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:208)
at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:202)
at org.openstack4j.openstack.compute.internal.ServerServiceImpl.boot(ServerServiceImpl.java:131)
at org.openstack4j.openstack.compute.internal.ServerServiceImpl.bootAndWaitActive(ServerServiceImpl.java:139)
...
Locked ownable synchronizers:
- <0x5084265d> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Dozens of other threads like this blocked because of it:
"jenkins.util.Timer [#2]" #48 daemon prio=5
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2e9dbd87> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:138)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.openstack4j.connectors.httpclient.HttpCommand.execute(HttpCommand.java:112)
at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.invokeRequest(HttpExecutorServiceImpl.java:65)
at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.invoke(HttpExecutorServiceImpl.java:56)
at org.openstack4j.connectors.httpclient.HttpExecutorServiceImpl.execute(HttpExecutorServiceImpl.java:32)
at org.openstack4j.core.transport.internal.HttpExecutor.execute(HttpExecutor.java:51)
at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:208)
at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:202)
at org.openstack4j.openstack.compute.internal.ServerServiceImpl.list(ServerServiceImpl.java:97)
at org.openstack4j.openstack.compute.internal.ServerServiceImpl.list(ServerServiceImpl.java:82)
...
Locked ownable synchronizers:
- <0x8691939> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x69b19783> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Switching to okhttp connector seem to resolve the problem for now. I hope I am not too optimistic to believe the ConnectionException{message=timeout, status=0}
s thrown are the same underlying problem handled better.
We are using 2.10 release with httpclient and facing issue with connection pool.
We are doing multiple server snapshot downloads in a sequence. First download works well but later downloads get stuck. As by default the max connections per route is 2; when increased withMaxConnectionsPerRoute , we could do more successful calls. But ultimately it gets stuck again after the connections are exhausted. It seems connections are open and not getting closed.
Config config=Config.newConfig().withMaxConnectionsPerRoute(4)
Logs where it gets stuck are -
2016/02/23 08:47:41:573 UTC [DEBUG] PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://endpoint:5000] can be kept alive indefinitely 2016/02/23 08:47:41:573 UTC [DEBUG] PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://endpoint:5000][total kept alive: 2; route allocated: 1 of 2; total allocated: 4 of 20] 2016/02/23 08:47:41:574 UTC [DEBUG] RequestAddCookies - CookieSpec selected: best-match 2016/02/23 08:47:41:574 UTC [DEBUG] RequestAuthCache - Auth cache not set in the context 2016/02/23 08:47:41:574 UTC [DEBUG] PoolingHttpClientConnectionManager - Connection request: [route: {}->http://endpoint:8774][total kept alive: 2; route allocated: 2 of 2; total allocated: 4 of 20]
Our download code makes sure that the input stream that we receive is closed. However, all operations from ServerService get stuck after the last successful download, however other calls from ImageService can be made.