I'm seeing this happening arbitrarily on some search requests. The same term will show up working a minute later though.
INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,368 CassandraIndexManager.java (line 318) ShardInfo for 3fae2fd4-dda2-47de-8a21-ecda58fa59c2-dev has expired
INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,370 CassandraIndexManager.java (line 388) Found reserved shard0(134958317770117392266381135524828078622):49548 TO 65536
INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,371 CassandraIndexManager.java (line 388) Found reserved shard1(134958317770117392266381135524828078622):49548 TO 65536
INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,371 CassandraIndexManager.java (line 388) Found reserved shard2(134958317770117392266381135524828078622):49548 TO 65536
INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,371 CassandraIndexManager.java (line 388) Found reserved shard3(134958317770117392266381135524828078622):49547 TO 65536
INFO [pool-7-thread-227] 2011-08-18 01:05:24,432 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-227] 2011-08-18 01:05:24,433 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-228] 2011-08-18 01:05:24,432 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-228] 2011-08-18 01:05:24,433 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-229] 2011-08-18 01:05:24,434 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-229] 2011-08-18 01:05:24,434 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-230] 2011-08-18 01:05:24,435 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-230] 2011-08-18 01:05:24,435 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-228] 2011-08-18 01:05:45,472 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-228] 2011-08-18 01:05:45,473 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-229] 2011-08-18 01:05:45,474 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-229] 2011-08-18 01:05:45,474 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-230] 2011-08-18 01:05:45,475 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-230] 2011-08-18 01:05:45,475 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-227] 2011-08-18 01:05:45,476 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-227] 2011-08-18 01:05:45,476 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-228] 2011-08-18 01:06:06,512 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-228] 2011-08-18 01:06:06,513 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-227] 2011-08-18 01:06:06,512 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-227] 2011-08-18 01:06:06,513 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-229] 2011-08-18 01:06:06,514 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-229] 2011-08-18 01:06:06,514 HttpMethodDirector.java (line 445) Retrying request
INFO [pool-7-thread-230] 2011-08-18 01:06:06,515 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
INFO [pool-7-thread-230] 2011-08-18 01:06:06,515 HttpMethodDirector.java (line 445) Retrying request
ERROR [491335754@qtp-1728699027-57] 2011-08-18 01:06:27,554 SolrException.java (line 151) org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: java.net.ConnectException: Connection timed out
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:171)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:137)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.solr.client.solrj.SolrServerException: java.net.ConnectException: Connection timed out
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:483)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:421)
at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:393)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.(Socket.java:375)
at java.net.Socket.(Socket.java:249)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:427)
... 11 more
I'm seeing this happening arbitrarily on some search requests. The same term will show up working a minute later though.
INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,368 CassandraIndexManager.java (line 318) ShardInfo for 3fae2fd4-dda2-47de-8a21-ecda58fa59c2-dev has expired INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,370 CassandraIndexManager.java (line 388) Found reserved shard0(134958317770117392266381135524828078622):49548 TO 65536 INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,371 CassandraIndexManager.java (line 388) Found reserved shard1(134958317770117392266381135524828078622):49548 TO 65536 INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,371 CassandraIndexManager.java (line 388) Found reserved shard2(134958317770117392266381135524828078622):49548 TO 65536 INFO [491335754@qtp-1728699027-57] 2011-08-18 01:05:03,371 CassandraIndexManager.java (line 388) Found reserved shard3(134958317770117392266381135524828078622):49547 TO 65536 INFO [pool-7-thread-227] 2011-08-18 01:05:24,432 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-227] 2011-08-18 01:05:24,433 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-228] 2011-08-18 01:05:24,432 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-228] 2011-08-18 01:05:24,433 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-229] 2011-08-18 01:05:24,434 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-229] 2011-08-18 01:05:24,434 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-230] 2011-08-18 01:05:24,435 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-230] 2011-08-18 01:05:24,435 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-228] 2011-08-18 01:05:45,472 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-228] 2011-08-18 01:05:45,473 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-229] 2011-08-18 01:05:45,474 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-229] 2011-08-18 01:05:45,474 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-230] 2011-08-18 01:05:45,475 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-230] 2011-08-18 01:05:45,475 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-227] 2011-08-18 01:05:45,476 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-227] 2011-08-18 01:05:45,476 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-228] 2011-08-18 01:06:06,512 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-228] 2011-08-18 01:06:06,513 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-227] 2011-08-18 01:06:06,512 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-227] 2011-08-18 01:06:06,513 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-229] 2011-08-18 01:06:06,514 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-229] 2011-08-18 01:06:06,514 HttpMethodDirector.java (line 445) Retrying request INFO [pool-7-thread-230] 2011-08-18 01:06:06,515 HttpMethodDirector.java (line 439) I/O exception (java.net.ConnectException) caught when processing request: Connection timed out INFO [pool-7-thread-230] 2011-08-18 01:06:06,515 HttpMethodDirector.java (line 445) Retrying request ERROR [491335754@qtp-1728699027-57] 2011-08-18 01:06:27,554 SolrException.java (line 151) org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: java.net.ConnectException: Connection timed out at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:171) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252) at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:137) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: org.apache.solr.client.solrj.SolrServerException: java.net.ConnectException: Connection timed out at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:483) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244) at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:421) at org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:393) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.ConnectException: Connection timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at java.net.Socket.(Socket.java:375)
at java.net.Socket.(Socket.java:249)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:427)
... 11 more
INFO [491335754@qtp-1728699027-57] 2011-08-18 01:06:27,555 SolrCore.java (line 1378) [3fae2fd4-dda2-47de-8a21-ecda58fa59c2-dev] webapp=/solandra path=/select params={spellcheck=true&fl=id,entity_id,entity_type,bundle,bundle_name,label,is_comment_count,ds_created,ds_changed,score,path,url,is_uid,tos_name&start=0&q=night&spellcheck.q=night&qf=content^40&hl.fl=content&json.nl=map&wt=json&rows=10} status=500 QTime=84187