search

loklak / loklak_server

Distributed Open Source twitter and social media message search server that anonymously collects, shares, dumps and indexes data http://api.loklak.org
GNU Lesser General Public License v2.1
1.38k stars 222 forks source link

ElasticsearchIllegalStateException: Future got interrupted #129

Closed smokingwheels closed 8 years ago

smokingwheels commented 8 years ago

Decided to give my peer a workout. Found this in the log file. You can put this in the too hard basket.

org.elasticsearch.index.engine.IndexFailedEngineException: [messages][3] Index failed for [TWITTER#659795855377985536] at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:369) at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:511) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:196) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.packed.PackedLongValues$Builder.(PackedLongValues.java:185) at org.apache.lucene.util.packed.DeltaPackedLongValues$Builder.(DeltaPackedLongValues.java:59) at org.apache.lucene.util.packed.PackedLongValues.deltaPackedBuilder(PackedLongValues.java:55) at org.apache.lucene.util.packed.PackedLongValues.deltaPackedBuilder(PackedLongValues.java:60) at org.apache.lucene.index.NumericDocValuesWriter.(NumericDocValuesWriter.java:44) at org.apache.lucene.index.DefaultIndexingChain$PerField.finish(DefaultIndexingChain.java:578) at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:324) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:241) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1526) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1252) at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:432) at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:364) ... 8 more Exception in thread "Thread-5484" org.elasticsearch.action.NoShardAvailableActionException: [messages][3] null at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.perform(TransportShardSingleOperationAction.java:175) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.onFailure(TransportShardSingleOperationAction.java:162) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.access$1300(TransportShardSingleOperationAction.java:116) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:197) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.index.IndexShardMissingException: [messages][3] missing at org.elasticsearch.index.IndexService.shardSafe(IndexService.java:210) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:98) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:194) ... 3 more 2015-10-29 18:18:26.996:INFO::pool-1-thread-3: /api/search.rss?q=Ellen+DeGeneres -> 100 records returned, 0 new 2015-10-29 18:18:27.789:INFO::Thread-5498: /api/search.rss scraping with query: Cristiano Ronaldo 2015-10-29 18:18:30.270:INFO::pool-1-thread-7: /api/search.rss?q=youtube -> 100 records returned, 100 new 2015-10-29 18:18:31.101:INFO::Thread-5502: /api/search.rss scraping with query: BillGates 2015-10-29 18:18:36.949:INFO::pool-1-thread-4: /api/search.rss?q=BillGates -> 44 records returned, 0 new 2015-10-29 18:18:37.776:INFO::Thread-5505: /api/search.rss scraping with query: BrunoMars 2015-10-29 18:18:39.540:INFO::pool-1-thread-9: /api/search.rss?q=Cristiano+Ronaldo -> 100 records returned, 0 new 2015-10-29 18:18:40.344:INFO::pool-1-thread-8: /api/search.rss?q=BrunoMars -> 38 records returned, 0 new 2015-10-29 18:18:40.345:INFO::Thread-5507: /api/search.rss scraping with query: instagram 2015-10-29 18:18:41.232:INFO::Thread-5509: /api/search.rss scraping with query: KingJames Exception in thread "Thread-5509" org.elasticsearch.action.NoShardAvailableActionException: [messages][3] null at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.perform(TransportShardSingleOperationAction.java:175) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.onFailure(TransportShardSingleOperationAction.java:162) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.access$1300(TransportShardSingleOperationAction.java:116) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:197) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: [messages][3] CurrentState[RECOVERING] operations only allowed when started/relocated at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1004) at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:992) at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:585) at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:176) at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:104) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:104) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:194) ... 3 more 2015-10-29 18:18:43.050:INFO::pool-1-thread-7: /api/search.rss?q=KingJames -> 21 records returned, 0 new 2015-10-29 18:18:43.042:INFO::pool-1-thread-3: /api/search.rss?q=instagram -> 100 records returned, 0 new 2015-10-29 18:18:43.941:INFO::Thread-5511: /api/search.rss scraping with query: Real_Liam_Payne 2015-10-29 18:18:44.961:INFO::Thread-5513: /api/search.rss scraping with query: KimKardashian Exception in thread "Thread-5511" org.elasticsearch.action.NoShardAvailableActionException: [messages][3] null at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.perform(TransportShardSingleOperationAction.java:175) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.onFailure(TransportShardSingleOperationAction.java:162) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.access$1300(TransportShardSingleOperationAction.java:116) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:197) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: [messages][3] CurrentState[RECOVERING] operations only allowed when started/relocated at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1004) at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:992) at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:585) at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:176) at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:104) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:104) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:194) ... 3 more 2015-10-29 18:18:45.051:INFO::pool-1-thread-4: /api/search.rss?q=Real_Liam_Payne -> 100 records returned, 0 new 2015-10-29 18:18:45.909:INFO::Thread-5515: /api/search.rss scraping with query: instagram 2015-10-29 18:18:47.665:INFO::pool-1-thread-8: /api/search.rss?q=KimKardashian -> 100 records returned, 0 new 2015-10-29 18:18:48.463:INFO::Thread-5517: /api/search.rss scraping with query: Kim Kardashian West 2015-10-29 18:18:48.506:INFO::pool-1-thread-7: /api/search.rss?q=instagram -> 100 records returned, 0 new 2015-10-29 18:18:49.328:INFO::Thread-5519: /api/search.rss scraping with query: pitbull 2015-10-29 18:18:56.839:INFO::pool-1-thread-6: /api/search.rss?q=pitbull -> 100 records returned, 0 new 2015-10-29 18:18:57.612:INFO::pool-1-thread-10: /api/search.rss?q=Kim+Kardashian+West -> 100 records returned, 0 new 2015-10-29 18:18:57.635:INFO::Thread-5521: /api/search.rss scraping with query: MileyCyrus 2015-10-29 18:18:59.184:INFO::Thread-5523: /api/search.rss scraping with query: youtube 2015-10-29 18:19:14.523:INFO::pool-1-thread-5: /api/search.rss?q=youtube -> 100 records returned, 0 new 2015-10-29 18:19:16.033:INFO::pool-1-thread-9: /api/search.rss?q=MileyCyrus -> 100 records returned, 0 new 2015-10-29 18:19:17.535:INFO::Thread-5526: /api/search.rss scraping with query: shakira 2015-10-29 18:19:20.696:INFO::Thread-5528: /api/search.rss scraping with query: youtube 2015-10-29 18:19:54.346:INFO::pool-1-thread-3: /api/search.rss?q=shakira -> 100 records returned, 0 new 2015-10-29 18:20:00.336:INFO::Thread-5531: /api/search.rss scraping with query: selenagomez 2015-10-29 18:21:46.713:INFO::pool-1-thread-7: /api/search.rss?q=youtube -> 100 records returned, 0 new 2015-10-29 18:23:13.355:WARN::pool-1-thread-9: Java heap space java.lang.OutOfMemoryError: Java heap space 2015-10-29 18:23:14.080:INFO::Thread-5533: /api/search.rss scraping with query: Louis_Tomlinson 2015-10-29 18:23:14.090:INFO::Thread-23: catched main termination signal 2015-10-29 18:23:22.499:INFO:oejs.ServerConnector:Thread-20: Stopped httpd:80@6104bc77{HTTP/1.1}{0.0.0.0:80} 2015-10-29 18:23:22.501:INFO:oejsh.ContextHandler:Thread-20: Stopped o.e.j.s.ServletContextHandler@11a78ca7{/,null,UNAVAILABLE} 2015-10-29 18:23:22.505:WARN::pool-1-thread-5: Future got interrupted org.elasticsearch.ElasticsearchIllegalStateException: Future got interrupted at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:47) at org.loklak.data.DAO$SearchLocalMessages.(DAO.java:627) at org.loklak.api.server.SearchServlet.doGet(SearchServlet.java:126) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:497) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:996) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:278) at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:117) at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:45) at org.loklak.data.DAO$SearchLocalMessages.(DAO.java:627) at org.loklak.api.server.SearchServlet.doGet(SearchServlet.java:126) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:497) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-10-29 18:23:22.509:INFO::Thread-23: catched caretaker termination signal 2015-10-29 18:23:22.509:INFO::Thread-23: closing DAO 2015-10-29 18:23:23.312:INFO::main: server terminated Exception in thread "Thread-5531" org.elasticsearch.action.NoShardAvailableActionException: [messages][5] null at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.perform(TransportShardSingleOperationAction.java:175) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.onFailure(TransportShardSingleOperationAction.java:162) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.access$1300(TransportShardSingleOperationAction.java:116) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:197) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.index.engine.EngineClosedException: [messages][5] CurrentState[CLOSED] at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:289) at org.elasticsearch.index.engine.InternalEngine.get(InternalEngine.java:224) at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:586) at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:176) at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:104) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:104) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44) at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:194) ... 3 more

Orbiter commented 8 years ago

well it looks like the index files are broken. I cannot find something that can be done programatically.