search

loklak / loklak_server

Distributed Open Source twitter and social media message search server that anonymously collects, shares, dumps and indexes data http://api.loklak.org
GNU Lesser General Public License v2.1
1.38k stars 222 forks source link

I have increased the JVM to 384 Meg. After some time I get java.lang.OutOfMemoryError: Java heap space #132

Closed smokingwheels closed 8 years ago

smokingwheels commented 8 years ago

I am running a list of wget commands to my server with no pause. Works fine on short runs. I will increase the JVM to 512 Mb and test tomorrow.

2015-11-09 14:26:22.574:INFO::Thread-19154: /api/search.rss scraping with query: @juanes 2015-11-09 14:26:43.076:INFO::pool-1-thread-9: /api/search.rss?q=%40juanes%0D -> 100 records returned, 0 new 2015-11-09 14:27:14.228:INFO::Thread-19156: /api/search.rss scraping with query: @juanes Exception in thread "Thread-17" java.lang.OutOfMemoryError: Java heap space at java.lang.Long.toUnsignedString(Long.java:242) at java.lang.Long.toHexString(Long.java:177) at org.loklak.data.Timeline.addTweet(Timeline.java:135) at org.loklak.data.Timeline.putAll(Timeline.java:159) at org.loklak.data.DAO.takeTimelineMax(DAO.java:484) at org.loklak.data.DAO.takeTimelineMin(DAO.java:467) at org.loklak.Caretaker.run(Caretaker.java:88) [messages][[messages][6]] IndexFailedEngineException[Index failed for [TWITTER#662317775449387008]]; nested: OutOfMemoryError[Java heap space]; at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:452) at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:815) at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1073) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:170) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.ByteBlockPool$DirectTrackingAllocator.getByteBlock(ByteBlockPool.java:103) at org.apache.lucene.util.ByteBlockPool.nextBuffer(ByteBlockPool.java:202) at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:287) at org.apache.lucene.index.SortedSetDocValuesWriter.addOneValue(SortedSetDocValuesWriter.java:120) at org.apache.lucene.index.SortedSetDocValuesWriter.addValue(SortedSetDocValuesWriter.java:84) at org.apache.lucene.index.DefaultIndexingChain.indexDocValue(DefaultIndexingChain.java:448) at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:376) at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1142) at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:518) at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:447) ... 10 more 2015-11-09 14:27:57.748:INFO::pool-1-thread-10: /api/search.rss?q=%40juanes%0D -> 100 records returned, 0 new 2015-11-09 14:27:58.574:INFO::Thread-19158: /api/search.rss scraping with query: @juanes 2015-11-09 14:28:08.644:INFO::pool-1-thread-6: /api/search.rss?q=%40juanes%0D -> 100 records returned, 0 new 2015-11-09 14:28:10.232:INFO::Thread-19160: /api/search.rss scraping with query: @Montanertwiter 2015-11-09 14:28:15.232:INFO::pool-1-thread-7: /api/search.rss?q=%40Montanertwiter%0D -> 1 records returned, 0 new 2015-11-09 14:28:27.768:INFO::Thread-19162: /api/search.rss scraping with query: @juanes [users][[users][4]] IndexFailedEngineException[Index failed for [TWITTER#alma_su_morales]]; nested: OutOfMemoryError[Java heap space]; at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:452) at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:815) at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1073) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:170) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.packed.PackedLongValues$Builder.(PackedLongValues.java:192) at org.apache.lucene.util.packed.PackedLongValues.packedBuilder(PackedLongValues.java:46) at org.apache.lucene.util.packed.PackedLongValues.packedBuilder(PackedLongValues.java:51) at org.apache.lucene.index.SortedSetDocValuesWriter.(SortedSetDocValuesWriter.java:60) at org.apache.lucene.index.DefaultIndexingChain.indexDocValue(DefaultIndexingChain.java:446) at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:376) at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363) at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:529) at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:447) ... 10 more NoShardAvailableActionException[No shard available for [get [users][_all][fansrosanachile]: routing [null]]]; nested: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: ShardNotFoundException[no such shard]; at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240) at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821) at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799) at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: ShardNotFoundException[no such shard]; Caused by: [users][[users][4]] ShardNotFoundException[no such shard] at org.elasticsearch.index.IndexService.shardSafe(IndexService.java:198) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [messages][[messages][5]] IndexFailedEngineException[Index failed for [TWITTER#663685005646917632]]; nested: OutOfMemoryError[Java heap space]; at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:452) at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:815) at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1073) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:170) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space Exception in thread "Thread-19085" NoShardAvailableActionException[No shard available for [get [messages][_all][663718024315191296]: routing [null]]]; nested: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: ShardNotFoundException[no such shard]; at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240) at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821) at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799) at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: ShardNotFoundException[no such shard]; Caused by: [messages][[messages][5]] ShardNotFoundException[no such shard] at org.elasticsearch.index.IndexService.shardSafe(IndexService.java:198) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-11-09 14:30:43.769:INFO::pool-1-thread-9: /api/search.rss?q=%40juanes%0D -> 100 records returned, 0 new Exception in thread "Thread-19061" NoShardAvailableActionException[No shard available for [get [messages][_all][663684626448297984]: routing [null]]]; nested: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: ShardNotFoundException[no such shard]; at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240) at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821) at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799) at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: ShardNotFoundException[no such shard]; Caused by: [messages][[messages][5]] ShardNotFoundException[no such shard] at org.elasticsearch.index.IndexService.shardSafe(IndexService.java:198) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-11-09 14:30:44.559:INFO::Thread-19164: /api/search.rss scraping with query: @juanes Exception in thread "Thread-19088" NoShardAvailableActionException[No shard available for [get [messages][_all][663718024315191296]: routing [null]]]; nested: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]; at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240) at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821) at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799) at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]; Caused by: [messages][[messages][5]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]] at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:957) at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:589) at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:156) at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:86) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:101) at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Exception in thread "Thread-19090" NoShardAvailableActionException[No shard available for [get [messages][_all][663718024315191296]: routing [null]]]; nested: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]; at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240) at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821) at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799) at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: RemoteTransportException[[Libra][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];

smokingwheels commented 8 years ago

Has been set to 600 Mb have not had error again.