Closed jbcpollak closed 10 years ago
Just an FYI, this is happening pretty frequently today, here is another stack trace (pretty similar to the previous one). Is there an segment.io outage or something?
[myproject segment.io failure websserver2]: <11>Mar 04 20:47:54 hostname [Thread-9] analytics Failed analytics response.Read timed outjava.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
at sun.reflect.GeneratedMethodAccessor381.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.http.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138)
at com.sun.proxy.$Proxy57.receiveResponseHeader(Unknown Source)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at com.github.segmentio.request.BlockingRequester.executeRequest(BlockingRequester.java:118)
at com.github.segmentio.request.BlockingRequester.send(BlockingRequester.java:60)
at com.github.segmentio.flush.Flusher.run(Flusher.java:91)
A very similar issue, when there is an error there is no retry causing lost of events.
2014-02-26 17:14:12,852 - [ERROR] - from analytics in Thread-7 Failed analytics response.The target server failed to respond org.apache.http.NoHttpResponseException: The target server failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) ~[org.apache.httpcomponents.httpcore-4.2.4.jar:4.2.4] at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) ~[org.apache.httpcomponents.httpcore-4.2.4.jar:4.2.4] at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) ~[org.apache.httpcomponents.httpcore-4.2.4.jar:4.2.4] at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) ~[org.apache.httpcomponents.httpcore-4.2.4.jar:4.2.4] at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:716) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:521) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) ~[org.apache.httpcomponents.httpclient-4.2.4.jar:4.2.4] at com.github.segmentio.request.BlockingRequester.send(BlockingRequester.java:63) ~[com.github.segmentio.analytics-0.3.1.jar:na] at com.github.segmentio.flush.Flusher.run(Flusher.java:91) [com.github.segmentio.analytics-0.3.1.jar:na]
just released in 0.4.2 :) let me know what you think!
HTTP timeouts were added in Pull Request https://github.com/segmentio/analytics-java/pull/11, and were released in 0.4.0. This was a big improvement because prior to that, requests would block indefinitely, and eventually all segment.io traffic would stop without warning.
Now with the timeouts individual requests can fail, but the overall system does not lock up.
The problem is when the requests fail, we now get exceptions in our logs (below), and the request is not retried. This means the data in the request is lost, and segment.io will not receive the events we are trying to submit.