Consistent exception on /ping

[GoogleCodeExporter]

Hi,

I am running an esemble of 6 clusters, each with a active-passive master pair 
and two servers. I am getting a lot these exceptions. I have monitored network 
connectivity between the hosts and it is fine.

04:00:59.658Server9740 - 29006007@qtp-28393594-1027 - 
org.jboss.resteasy.core.SynchronousDispatcher - Failed executing GET /ping
org.jboss.resteasy.spi.WriterException: org.mortbay.jetty.EofException
        at org.jboss.resteasy.core.ServerResponse.writeTo(ServerResponse.java:233) ~[resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.core.SynchronousDispatcher.writeJaxrsResponse(SynchronousDispatcher.java:563) [resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:484) [resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:207) [resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.1.0.GA.jar:na]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) [servlet-api-2.5-20081211.jar:na]
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.Server.handle(Server.java:326) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) [jetty-util-6.1.22.jar:6.1.22]
Caused by: org.mortbay.jetty.EofException: null
        at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787) ~[jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpGenerator.complete(HttpGenerator.java:674) ~[jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpConnection.commitResponse(HttpConnection.java:653) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpConnection$Output.close(HttpConnection.java:990) ~[jetty-6.1.22.jar:6.1.22]
        at org.codehaus.jackson.impl.Utf8Generator.close(Utf8Generator.java:1026) ~[jackson-core-asl-1.7.2.jar:1.7.2]
        at terrastore.util.json.JsonUtils.write(JsonUtils.java:168) ~[terrastore-0.8.2-SNAPSHOT.jar:na]
        at terrastore.server.impl.support.JsonValuesProvider.writeTo(JsonValuesProvider.java:49) ~[terrastore-0.8.2-SNAPSHOT.jar:na]
        at terrastore.server.impl.support.JsonValuesProvider.writeTo(JsonValuesProvider.java:39) ~[terrastore-0.8.2-SNAPSHOT.jar:na]
        at org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proceed(MessageBodyWriterContextImpl.java:117) ~[resteasy-jaxrs-2.1.0.GA.jar:na]
        at terrastore.server.impl.cors.CorsInterceptor.write(CorsInterceptor.java:106) ~[terrastore-0.8.2-SNAPSHOT.jar:na]
        at org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proceed(MessageBodyWriterContextImpl.java:123) ~[resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor.write(GZIPEncodingInterceptor.java:48) ~[resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proceed(MessageBodyWriterContextImpl.java:123) ~[resteasy-jaxrs-2.1.0.GA.jar:na]
        at org.jboss.resteasy.core.ServerResponse.writeTo(ServerResponse.java:222) ~[resteasy-jaxrs-2.1.0.GA.jar:na]
        ... 19 common frames omitted
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_26]
        at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_26]
        at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_26]
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_26]
        at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_26]
        at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:232) ~[jetty-6.1.22.jar:6.1.22]
        at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:211) [jetty-6.1.22.jar:6.1.22]
        at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:710) ~[jetty-6.1.22.jar:6.1.22]
        ... 32 common frames omitted

Original issue reported on code.google.com by rohi...@gmail.com on 29 Jul 2011 at 10:49

[GoogleCodeExporter]

It seems the connection has been closed by the remote client (crash, timeout, 
transient network burp...), does it happen only with an ensemble setup and/or a 
given number of clusters?

Also, why are you constantly GETting the /ping bucket?

Original comment by sergio.b...@gmail.com on 30 Jul 2011 at 8:36

[GoogleCodeExporter]

I am seeing this exception across the ensemble, in each of the cluster. I am 
using /ping to check the health of the cluster. So your take is that my host, 
which is doing GET /ping is closing the connection? Is there a better way to do 
it?

Original comment by rohi...@gmail.com on 30 Jul 2011 at 2:33

[GoogleCodeExporter]

It seems so: Jetty tries to flush the response (at 
org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:710) 
~[jetty-6.1.22.jar:6.1.22]), but finds the connection has been reset.

BTW, cluster health could be checked via cluster statistics: 
http://code.google.com/p/terrastore/wiki/HTTP_Client_API#Cluster_Statistics

Let me know.

Original comment by sergio.b...@gmail.com on 31 Jul 2011 at 10:08

[GoogleCodeExporter]

Original comment by sergio.b...@gmail.com on 2 Aug 2011 at 3:42

• Changed state: Invalid