search

loklak / loklak_server

Distributed Open Source twitter and social media message search server that anonymously collects, shares, dumps and indexes data http://api.loklak.org
GNU Lesser General Public License v2.1
1.38k stars 222 forks source link

java.lang.OutOfMemoryError: Java heap space with JVM set at 650 mb on 1 Gb peer. #146

Closed smokingwheels closed 8 years ago

smokingwheels commented 8 years ago

My Loklak server has been crashing after a 1 hour started about 7 days ago. Also have failed message import on my peer because the imported messages have not showed up in the dump page on the webserver. I think this is due to hdd space running out on my server

Sorry for the long post but github is rejecting Winrar Zip format.

2015-12-09 22:36:06.805:INFO::pool-2-thread-1: retrieval of 20 new messages for q = mint, pushed to backend synchronously in 444 ms; pendingQueries = 0, pendingContext = 13, harvestedContext = 3
2015-12-09 22:36:11.842:INFO::pool-2-thread-1: retrieval of 14 new messages for q = expert, pushed to backend synchronously in 233 ms; pendingQueries = 0, pendingContext = 21, harvestedContext = 4
2015-12-09 22:36:17.157:INFO::Thread-27: retrieval of 5 new messages for q = jeditraining, scheduled push; pendingQueries = 0, pendingContext = 24, harvestedContext = 4
2015-12-09 22:36:20.395:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 5; max: 200]
2015-12-09 22:36:23.004:INFO::Thread-27: retrieval of 20 new messages for q = from:hubspot, scheduled push; pendingQueries = 0, pendingContext = 32, harvestedContext = 6
2015-12-09 22:36:28.545:INFO::Thread-27: retrieval of 20 new messages for q = from:mint_zero02, scheduled push; pendingQueries = 0, pendingContext = 47, harvestedContext = 7
2015-12-09 22:36:32.522:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 5; max: 200]
2015-12-09 22:36:34.517:INFO::Thread-27: retrieval of 9 new messages for q = trialsofthetemple, scheduled push; pendingQueries = 0, pendingContext = 46, harvestedContext = 9
2015-12-09 22:36:38.948:INFO::Thread-27: retrieval of 10 new messages for q = from: , scheduled push; pendingQueries = 0, pendingContext = 48, harvestedContext = 10
2015-12-09 22:36:40.110:INFO::Thread-27: retrieval of 3 new messages for q = from:technology, scheduled push; pendingQueries = 0, pendingContext = 48, harvestedContext = 11
2015-12-09 22:36:43.331:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 5; max: 200]
2015-12-09 22:36:45.396:INFO::Thread-27: retrieval of 20 new messages for q = from:HubSpotSupport, scheduled push; pendingQueries = 0, pendingContext = 60, harvestedContext = 13
2015-12-09 22:36:46.982:INFO::Thread-27: retrieval of 14 new messages for q = from:9GAG, scheduled push; pendingQueries = 0, pendingContext = 59, harvestedContext = 14
2015-12-09 22:36:49.432:INFO::pool-1-thread-9: /
2015-12-09 22:36:49.574:INFO::Thread-27: retrieval of 20 new messages for q = from:AceofSpadesHQ, scheduled push; pendingQueries = 0, pendingContext = 64, harvestedContext = 15
2015-12-09 22:36:51.434:INFO::Thread-27: retrieval of 14 new messages for q = from:WatchMirandaGo, scheduled push; pendingQueries = 0, pendingContext = 70, harvestedContext = 16
2015-12-09 22:36:52.320:INFO::Thread-27: retrieval of 5 new messages for q = from:HubSpotDev, scheduled push; pendingQueries = 0, pendingContext = 69, harvestedContext = 17
2015-12-09 22:36:55.492:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 5; max: 200]
2015-12-09 22:36:56.184:INFO::Thread-56: /api/search.json scraping with query: katyperry
2015-12-09 22:36:56.397:INFO::Thread-27: success pushing 207 messages to backend in 1st attempt in 904 ms
2015-12-09 22:36:58.721:INFO::Thread-27: retrieval of 2 new messages for q = from:CatullusB, scheduled push; pendingQueries = 0, pendingContext = 70, harvestedContext = 19
2015-12-09 22:37:02.985:INFO::Thread-27: retrieval of 1 new messages for q = from:nickrioux, scheduled push; pendingQueries = 0, pendingContext = 69, harvestedContext = 20
2015-12-09 22:37:18.579:INFO::Thread-27: retrieval of 14 new messages for q = darthvader, scheduled push; pendingQueries = 0, pendingContext = 75, harvestedContext = 21
2015-12-09 22:37:29.141:INFO::pool-1-thread-5: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 19 new
2015-12-09 22:37:34.368:INFO::Thread-27: retrieval of 20 new messages for q = from:rin_h_bot_, scheduled push; pendingQueries = 0, pendingContext = 82, harvestedContext = 22
2015-12-09 22:37:39.862:INFO::Thread-76: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 22:37:44.325:INFO::Thread-27: retrieval of 20 new messages for q = from:anaritafarrica, scheduled push; pendingQueries = 0, pendingContext = 87, harvestedContext = 23
2015-12-09 22:37:44.727:INFO::pool-1-thread-4: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 2 records returned, 0 new
2015-12-09 22:37:53.133:INFO::Thread-83: /api/search.json scraping with query: smokingwheels
2015-12-09 22:38:04.487:INFO::Thread-27: retrieval of 16 new messages for q = jedi, scheduled push; pendingQueries = 0, pendingContext = 95, harvestedContext = 24
2015-12-09 22:38:05.500:INFO::pool-1-thread-10: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 1 new
2015-12-09 22:38:11.970:INFO::Thread-27: retrieval of 20 new messages for q = from:daniellejdistef, scheduled push; pendingQueries = 0, pendingContext = 107, harvestedContext = 25
2015-12-09 22:38:16.695:INFO::Thread-27: retrieval of 20 new messages for q = from:leonawest_relax, scheduled push; pendingQueries = 0, pendingContext = 117, harvestedContext = 26
2015-12-09 22:38:24.699:INFO::Thread-27: retrieval of 20 new messages for q = from:miyu_cp_bot, scheduled push; pendingQueries = 0, pendingContext = 116, harvestedContext = 27
2015-12-09 22:38:27.994:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 9; max: 200]
2015-12-09 22:38:30.222:INFO::Thread-27: retrieval of 10 new messages for q = mindpower, scheduled push; pendingQueries = 0, pendingContext = 117, harvestedContext = 29
2015-12-09 22:38:35.384:INFO::Thread-27: retrieval of 15 new messages for q = from:sfnohohon, scheduled push; pendingQueries = 0, pendingContext = 122, harvestedContext = 30
2015-12-09 22:38:39.509:INFO::Thread-27: retrieval of 19 new messages for q = askbelieber, scheduled push; pendingQueries = 0, pendingContext = 121, harvestedContext = 31
2015-12-09 22:38:39.937:INFO::Thread-27: retrieval of 3 new messages for q = from:BrentC27, scheduled push; pendingQueries = 0, pendingContext = 121, harvestedContext = 32
2015-12-09 22:38:41.682:INFO::Thread-27: retrieval of 20 new messages for q = from:ALO_Leafa_nr, scheduled push; pendingQueries = 0, pendingContext = 125, harvestedContext = 33
2015-12-09 22:38:43.808:INFO::Thread-27: retrieval of 20 new messages for q = from:ryanbonnici, scheduled push; pendingQueries = 0, pendingContext = 151, harvestedContext = 34
2015-12-09 22:38:45.329:INFO::Thread-118: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 22:38:45.480:INFO::Thread-27: retrieval of 17 new messages for q = from:catduquesa, scheduled push; pendingQueries = 0, pendingContext = 150, harvestedContext = 35
2015-12-09 22:38:46.681:INFO::pool-1-thread-9: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:38:47.248:INFO::Thread-27: retrieval of 20 new messages for q = from:eli_univ, scheduled push; pendingQueries = 0, pendingContext = 165, harvestedContext = 36
2015-12-09 22:38:49.721:INFO::Thread-27: retrieval of 17 new messages for q = from:krispekitty, scheduled push; pendingQueries = 0, pendingContext = 168, harvestedContext = 37
2015-12-09 22:38:53.015:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 9; max: 200]
2015-12-09 22:38:54.235:INFO::Thread-27: success pushing 340 messages to backend in 1st attempt in 1217 ms
2015-12-09 22:38:57.315:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 9; max: 200]
2015-12-09 22:38:59.860:INFO::Thread-27: retrieval of 19 new messages for q = from:geek_freak_, scheduled push; pendingQueries = 0, pendingContext = 168, harvestedContext = 40
2015-12-09 22:39:00.585:INFO::Thread-27: retrieval of 2 new messages for q = from:dougdoesdenver, scheduled push; pendingQueries = 0, pendingContext = 167, harvestedContext = 41
2015-12-09 22:39:01.221:INFO::Thread-27: retrieval of 2 new messages for q = from:AbbeyRoad, scheduled push; pendingQueries = 0, pendingContext = 166, harvestedContext = 42
2015-12-09 22:39:03.383:INFO::Thread-27: retrieval of 20 new messages for q = from:managewski, scheduled push; pendingQueries = 0, pendingContext = 174, harvestedContext = 43
2015-12-09 22:39:03.810:INFO::Thread-27: retrieval of 2 new messages for q = from:ESGAdam, scheduled push; pendingQueries = 0, pendingContext = 173, harvestedContext = 44
2015-12-09 22:39:09.868:INFO::Thread-27: retrieval of 20 new messages for q = 2016, scheduled push; pendingQueries = 0, pendingContext = 186, harvestedContext = 45
2015-12-09 22:39:13.299:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 9; max: 200]
2015-12-09 22:39:16.377:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 9; max: 200]
2015-12-09 22:39:18.425:INFO::Thread-27: retrieval of 20 new messages for q = from:JeremyLittau, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 48
2015-12-09 22:39:20.730:INFO::Thread-27: retrieval of 20 new messages for q = from:Johnathan_FNAF, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 49
2015-12-09 22:39:21.678:INFO::Thread-27: retrieval of 6 new messages for q = from:glenn_purcell, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 50
2015-12-09 22:39:25.116:INFO::Thread-27: retrieval of 9 new messages for q = democracynow, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 51
2015-12-09 22:39:30.482:INFO::Thread-27: retrieval of 8 new messages for q = from:hollykhildreth, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 52
2015-12-09 22:39:38.642:INFO::Thread-27: retrieval of 20 new messages for q = sts, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 53
2015-12-09 22:39:40.026:INFO::Thread-27: retrieval of 8 new messages for q = from:auremoser, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 54
2015-12-09 22:39:45.698:INFO::Thread-27: retrieval of 20 new messages for q = from:TawanPKK, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 55
2015-12-09 22:39:48.760:INFO::Thread-27: retrieval of 20 new messages for q = from:heyheyman, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 56
2015-12-09 22:39:59.239:INFO::Thread-27: retrieval of 20 new messages for q = disneyland, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 57
2015-12-09 22:40:02.427:INFO::Thread-27: connection pool: [leased: 0; pending: 0; available: 9; max: 200]
2015-12-09 22:40:03.471:INFO::Thread-27: success pushing 246 messages to backend in 1st attempt in 1042 ms
2015-12-09 22:40:05.873:INFO::Thread-27: retrieval of 20 new messages for q = from:jedi_paige, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 58
2015-12-09 22:40:11.145:INFO::Thread-27: retrieval of 19 new messages for q = startwith1thing, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 59
2015-12-09 22:40:13.578:INFO::Thread-27: retrieval of 20 new messages for q = from:Mariasienna1221, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 60
2015-12-09 22:40:15.708:INFO::Thread-27: retrieval of 20 new messages for q = from:aiqrebelde, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 61
2015-12-09 22:40:17.431:INFO::Thread-27: retrieval of 12 new messages for q = from:jacktenz, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 62
2015-12-09 22:40:23.487:INFO::Thread-27: retrieval of 20 new messages for q = は行でわかるクリスマスの過ごし方, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 63
2015-12-09 22:40:26.875:INFO::Thread-27: retrieval of 20 new messages for q = from:nscalice, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 64
2015-12-09 22:40:33.881:INFO::Thread-27: retrieval of 16 new messages for q = worldnews, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 65
2015-12-09 22:40:37.288:INFO::Thread-27: retrieval of 20 new messages for q = from:umikiti_kotori, scheduled push; pendingQueries = 0, pendingContext = 200, harvestedContext = 66
2015-12-09 22:40:40.459:INFO::Thread-27: connection pool: [leased: 1; pending: 0; available: 7; max: 200]
2015-12-09 22:40:40.543:INFO::Thread-172: /api/search.json scraping with query: near:perth near:"Perth" within:68mi near:"Perth" within:68mi
2015-12-09 22:40:41.523:INFO::Thread-174: /api/search.json scraping with query: youtube
2015-12-09 22:43:52.951:INFO::Thread-184: /api/search.json scraping with query: smokingwheels
2015-12-09 22:44:42.909:INFO::Thread-189: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 22:45:16.559:INFO::Thread-192: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 22:45:40.928:INFO::Thread-194: /api/search.json scraping with query: youtube
[messages][[messages][6]] IndexFailedEngineException[Index failed for [TWITTER#674721523706408960]]; nested: OutOfMemoryError[Java heap space];
    at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:452)
    at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556)
    at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:815)
    at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1073)
    at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:170)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
2015-12-09 22:47:22.296:INFO::pool-1-thread-7: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:47:22.462:INFO::pool-1-thread-10: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 99 records returned, 0 new
2015-12-09 22:47:22.472:INFO::pool-1-thread-6: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 99 records returned, 0 new
2015-12-09 22:47:24.120:INFO::Thread-199: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 22:47:24.982:INFO::pool-1-thread-4: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 3 new
2015-12-09 22:47:27.665:INFO::pool-1-thread-7: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 84 records returned, 0 new
2015-12-09 22:47:35.464:INFO::pool-1-thread-9: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:47:42.120:INFO::Thread-201: /api/search.json scraping with query: youtube
2015-12-09 22:47:56.171:INFO::Thread-204: /api/search.json scraping with query: katyperry
2015-12-09 22:48:09.589:INFO::pool-1-thread-5: /api/search.json?q=near%3Aperth%2Bnear%3A%22Perth%22%2Bwithin%3A68mi&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 11 new
2015-12-09 22:50:01.495:WARN::pool-1-thread-3: Java heap space
java.lang.OutOfMemoryError: Java heap space
2015-12-09 22:50:06.993:INFO::Thread-209: /api/search.json scraping with query: mozilla
2015-12-09 22:50:07.574:INFO::Thread-210: /api/search.json scraping with query: smokingwheels
Exception in thread "Thread-201" [queries][[queries][4]] IndexFailedEngineException[Index failed for [TWITTER#youtube]]; nested: OutOfMemoryError[Java heap space];
    at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:452)
    at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556)
    at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:815)
    at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1073)
    at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:170)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
[users][[users][6]] IndexFailedEngineException[Index failed for [TWITTER#abckatyperryfr]]; nested: OutOfMemoryError[Java heap space];
    at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:452)
    at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556)
    at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:815)
    at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1073)
    at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:170)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
2015-12-09 22:50:58.166:INFO::Thread-213: /api/search.json scraping with query: katyperry
Exception in thread "Thread-213" NoShardAvailableActionException[No shard available for [get [messages][_all][674722827396780032]: routing [null]]]; nested: RemoteTransportException[[Baron Macabre][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: ShardNotFoundException[no such shard];
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799)
    at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: RemoteTransportException[[Baron Macabre][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: ShardNotFoundException[no such shard];
Caused by: [messages][[messages][3]] ShardNotFoundException[no such shard]
    at org.elasticsearch.index.IndexService.shardSafe(IndexService.java:198)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:95)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275)
    at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2015-12-09 22:51:09.426:INFO::Thread-216: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 22:51:30.476:INFO::pool-1-thread-9: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 95 records returned, 0 new
2015-12-09 22:51:32.887:INFO::pool-1-thread-4: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:51:36.730:WARN::pool-1-thread-7: No shard available for [get [users][_all][wadai4u]: routing [null]]
NoShardAvailableActionException[No shard available for [get [users][_all][wadai4u]: routing [null]]]; nested: RemoteTransportException[[Baron Macabre][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: EngineClosedException[CurrentState[CLOSED] Closed];
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799)
    at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: 
RemoteTransportException[[Baron Macabre][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: EngineClosedException[CurrentState[CLOSED] Closed];
Caused by: 
[users][[users][4]] EngineClosedException[CurrentState[CLOSED] Closed]
    at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:314)
    at org.elasticsearch.index.engine.InternalEngine.get(InternalEngine.java:309)
    at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:590)
    at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:156)
    at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:86)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:101)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275)
    at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2015-12-09 22:51:37.034:INFO::pool-1-thread-10: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:51:37.113:INFO::pool-1-thread-6: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:53:52.023:INFO::Thread-220: /api/search.json scraping with query: smokingwheels
2015-12-09 22:53:54.365:INFO::pool-1-thread-9: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:54:51.984:INFO::Thread-222: /api/search.json scraping with query: smokingwheels
2015-12-09 22:54:54.326:INFO::pool-1-thread-7: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:56:40.485:INFO::Thread-224: /api/search.json scraping with query: youtube
2015-12-09 22:56:42.829:INFO::pool-1-thread-8: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:58:40.450:INFO::Thread-226: /api/search.json scraping with query: youtube
2015-12-09 22:58:42.826:INFO::pool-1-thread-5: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:58:51.785:INFO::Thread-228: /api/search.json scraping with query: smokingwheels
2015-12-09 22:58:53.812:INFO::pool-1-thread-4: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 22:59:44.802:INFO::Thread-230: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 22:59:47.151:INFO::pool-1-thread-10: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 99 records returned, 0 new
2015-12-09 22:59:54.956:INFO::Thread-232: /api/search.json scraping with query: katyperry
2015-12-09 22:59:56.972:INFO::pool-1-thread-8: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:01:55.104:INFO::Thread-234: /api/search.json scraping with query: katyperry
2015-12-09 23:01:57.456:INFO::pool-1-thread-4: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:01:59.964:INFO::Thread-236: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 23:02:02.024:INFO::pool-1-thread-10: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 99 records returned, 0 new
2015-12-09 23:02:32.707:INFO::Thread-238: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 23:02:35.034:INFO::pool-1-thread-6: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 99 records returned, 0 new
2015-12-09 23:02:51.764:INFO::Thread-240: /api/search.json scraping with query: smokingwheels
2015-12-09 23:02:53.781:INFO::pool-1-thread-9: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:02:57.990:INFO::Thread-242: /api/search.json scraping with query: katyperry
2015-12-09 23:03:00.004:INFO::pool-1-thread-5: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:03:05.180:INFO::Thread-244: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 23:03:07.199:INFO::pool-1-thread-7: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 99 records returned, 0 new
2015-12-09 23:03:40.404:INFO::Thread-246: /api/search.json scraping with query: youtube
2015-12-09 23:03:42.733:INFO::pool-1-thread-3: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:04:51.904:INFO::Thread-248: /api/search.json scraping with query: smokingwheels
2015-12-09 23:04:54.227:INFO::pool-1-thread-9: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:05:39.229:INFO::Thread-250: /api/search.json scraping with query: near:perth near:"Perth" within:68mi near:"Perth" within:68mi
2015-12-09 23:05:40.917:INFO::Thread-252: /api/search.json scraping with query: youtube
2015-12-09 23:06:22.091:INFO::Thread-254: /api/search.json scraping with query: mozlando OR #mozlando include:retweets
2015-12-09 23:06:53.282:INFO::Thread-256: /api/search.json scraping with query: smokingwheels
2015-12-09 23:07:45.009:INFO::Thread-258: /api/search.json scraping with query: youtube
2015-12-09 23:08:01.567:INFO::pool-1-thread-6: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:08:01.568:INFO::pool-1-thread-3: /api/search.json?q=mozlando%2BOR%2B%23mozlando%2Binclude%3Aretweets&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 99 records returned, 0 new
2015-12-09 23:08:01.568:INFO::pool-1-thread-10: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:08:01.569:INFO::pool-1-thread-8: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:08:01.696:WARN::pool-1-thread-4: No shard available for [get [users][_all][georgieandthree]: routing [null]]
NoShardAvailableActionException[No shard available for [get [users][_all][georgieandthree]: routing [null]]]; nested: RemoteTransportException[[Baron Macabre][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: EngineClosedException[CurrentState[CLOSED] Closed];
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799)
    at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: 
RemoteTransportException[[Baron Macabre][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: EngineClosedException[CurrentState[CLOSED] Closed];
Caused by: 
[users][[users][4]] EngineClosedException[CurrentState[CLOSED] Closed]
    at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:314)
    at org.elasticsearch.index.engine.InternalEngine.get(InternalEngine.java:309)
    at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:590)
    at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:156)
    at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:86)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:101)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275)
    at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2015-12-09 23:08:59.747:INFO::Thread-260: /api/search.json scraping with query: Steven
2015-12-09 23:09:13.249:WARN::pool-1-thread-6: No shard available for [get [users][_all][nabilirfan97]: routing [null]]
NoShardAvailableActionException[No shard available for [get [users][_all][nabilirfan97]: routing [null]]]; nested: RemoteTransportException[[Baron Macabre][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: EngineClosedException[CurrentState[CLOSED] Closed];
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821)
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799)
    at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: 
RemoteTransportException[[Baron Macabre][localhost/127.0.0.1:9300][indices:data/read/get[s]]]; nested: EngineClosedException[CurrentState[CLOSED] Closed];
Caused by: 
[users][[users][4]] EngineClosedException[CurrentState[CLOSED] Closed]
    at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:314)
    at org.elasticsearch.index.engine.InternalEngine.get(InternalEngine.java:309)
    at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:590)
    at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:156)
    at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:86)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:101)
    at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282)
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275)
    at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2015-12-09 23:10:39.384:INFO::Thread-262: /api/search.json scraping with query: near:perth near:"Perth" within:68mi near:"Perth" within:68mi
2015-12-09 23:11:24.474:INFO::Thread-264: /api/search.json scraping with query: smokingwheels
2015-12-09 23:27:05.587:INFO::Thread-266: /api/search.json scraping with query: katyperry
2015-12-09 23:27:25.258:INFO::pool-1-thread-7: /api/search.json?q=near%3Aperth%2Bnear%3A%22Perth%22%2Bwithin%3A68mi&timezoneOffset=0&maximumRecords=100&source=all&minified=true&timeout=2000 -> 0 records returned, 0 new
2015-12-09 23:27:26.803:INFO::pool-1-thread-5: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:27:26.812:INFO::pool-1-thread-3: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:27:54.719:INFO::Thread-268: /api/search.json scraping with query: katyperry
2015-12-09 23:27:58.551:INFO::pool-1-thread-8: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:29:54.709:INFO::Thread-270: /api/search.json scraping with query: katyperry
2015-12-09 23:29:58.681:INFO::pool-1-thread-7: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:30:40.034:INFO::Thread-272: /api/search.json scraping with query: youtube
2015-12-09 23:30:43.673:INFO::pool-1-thread-9: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:32:51.538:INFO::Thread-274: /api/search.json scraping with query: smokingwheels
2015-12-09 23:32:54.471:INFO::pool-1-thread-4: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:34:40.007:INFO::Thread-276: /api/search.json scraping with query: youtube
2015-12-09 23:34:44.110:INFO::pool-1-thread-10: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:34:51.437:INFO::Thread-278: /api/search.json scraping with query: smokingwheels
2015-12-09 23:34:54.815:INFO::Thread-280: /api/search.json scraping with query: katyperry
2015-12-09 23:34:54.820:INFO::pool-1-thread-5: /api/search.json?q=smokingwheels&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:35:01.380:INFO::pool-1-thread-9: /api/search.json?q=katyperry&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:35:40.012:INFO::Thread-282: /api/search.json scraping with query: youtube
2015-12-09 23:35:48.313:INFO::pool-1-thread-6: /api/search.json?q=youtube&timezoneOffset=-480&maximumRecords=100&source=all&minified=true&timeout=2000 -> 100 records returned, 0 new
2015-12-09 23:36:51.403:INFO::Thread-284: /api/search.json scraping with query: smokingwheels
2015-12-09 23:36:54.726:INFO::Thread-286: /api/search.json scraping with query: katyperry
2015-12-09 23:37:21.180:WARN::pool-1-thread-10: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "Thread-18" java.lang.OutOfMemoryError: Java heap space

Last message in the log file after it crashes

DengYiping commented 8 years ago

Please put your dump file into code block like this so that this ticket looks good. I will try to duplicate your error. And please provides your information of your environment, like your operating system. Try to run test on your machine.

smokingwheels commented 8 years ago

I have reinstalled on VM with a HDD and disabled updates has been working fine for 3 weeks. My system was Debian 8 64 bit on a SSD drive.