Out of heap memory java issue while using restore command.

[starit1977]

Hi,

We use - 1) elastic search version 0.90.3, 2) Plugin version was picked up from website. 3) Heap space size is default i.e. 256M 4) Data size is 46MB 5) Number of data files - Not sure how do i check this? 6) Number of nodes - 1 7) Index settings - Settings - {"test":{"settings":{"index.number_of_shards":"5","index.number_of_replicas":"1","index.version.created":"900399"}}} Mappings - {"test":{"export":{"properties":{"compression":{"type":"string"},"fields":{"type":"string"},"output_file":{"type":"string"}}},"device":{"properties":{"deviceId":{"type":"string"},"packages":{"properties":{"dependencies":{"properties":{"id":{"type":"string"},"version":{"type":"string"}}},"id":{"type":"string"},"source":{"type":"string"},"version":{"type":"string"}}},"properties":{"properties":{"pin":{"properties":{"source":{"type":"string"},"value":{"type":"string"}}}}},"supportedAPIs":{"properties":{"id":{"type":"string"},"source":{"type":"string"},"type":{"type":"string"},"version":{"type":"string"}}},"ts":{"type":"long"},"vendorId":{"type":"string"}}}}}

The index name is test and it has around 45 MB worth of data. The dump worked perfectly fine. However, when i use restore it gave me out of heap error.

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/data/elasticsearch/nodes/0/dump$ local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/data/elasticsearch/nodes/0/dump$ local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/data/elasticsearch/nodes/0/dump$ curl -X POST 'http://localhost:9200/test/_restore' [2013-09-17 17:16:50,035][INFO ][cluster.metadata ] [Eric] [test] creating index, cause [api], shards [5]/[1], mappings [] [2013-09-17 17:16:50,076][DEBUG][index.gateway ] [Eric] [test][0] starting recovery from local ... [2013-09-17 17:16:50,082][DEBUG][index.gateway ] [Eric] [test][1] starting recovery from local ... [2013-09-17 17:16:50,084][DEBUG][index.gateway ] [Eric] [test][0] recovery completed from local, took [8ms] index : files [0] with total_size [0b], took[0s] : recovered_files [0] with total_size [0b] : reusing_files [0] with total_size [0b] start : took [8ms], check_index [0s] translog : number_of_operations [0], took [0s] [2013-09-17 17:16:50,092][DEBUG][index.gateway ] [Eric] [test][1] recovery completed from local, took [10ms] index : files [0] with total_size [0b], took[0s] : recovered_files [0] with total_size [0b] : reusing_files [0] with total_size [0b] start : took [10ms], check_index [0s] translog : number_of_operations [0], took [0s] [2013-09-17 17:16:50,101][DEBUG][index.gateway ] [Eric] [test][2] starting recovery from local ... [2013-09-17 17:16:50,107][INFO ][cluster.metadata ] [Eric] [test] create_mapping [export] [2013-09-17 17:16:50,108][DEBUG][index.gateway ] [Eric] [test][2] recovery completed from local, took [7ms] index : files [0] with total_size [0b], took[1ms] : recovered_files [0] with total_size [0b] : reusing_files [0] with total_size [0b] start : took [6ms], check_index [0s] translog : number_of_operations [0], took [0s] [2013-09-17 17:16:50,130][DEBUG][index.gateway ] [Eric] [test][3] starting recovery from local ... [2013-09-17 17:16:50,138][DEBUG][index.gateway ] [Eric] [test][4] starting recovery from local ... [2013-09-17 17:16:50,139][DEBUG][index.gateway ] [Eric] [test][3] recovery completed from local, took [10ms] index : files [0] with total_size [0b], took[0s] : recovered_files [0] with total_size [0b] : reusing_files [0] with total_size [0b] start : took [9ms], check_index [0s] translog : number_of_operations [0], took [0s] [2013-09-17 17:16:50,147][DEBUG][index.gateway ] [Eric] [test][4] recovery completed from local, took [9ms] index : files [0] with total_size [0b], took[0s] : recovered_files [0] with total_size [0b] : reusing_files [0] with total_size [0b] start : took [8ms], check_index [0s] translog : number_of_operations [0], took [0s] [2013-09-17 17:16:50,163][INFO ][cluster.metadata ] [Eric] [test] create_mapping [device] [2013-09-17 17:17:00,770][WARN ][index.engine.robin ] [Eric] [test][2] failed engine java.lang.OutOfMemoryError: Java heap space [2013-09-17 17:17:05,837][DEBUG][action.bulk ] [Eric] [test][2] failed to execute bulk item (index) index {[test][device][1336553], source[{"data......................"}]} org.elasticsearch.index.engine.IndexFailedEngineException: [test][2] Index failed for [device#1336553] at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:503) at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:341) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:397) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:521) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.apache.lucene.store.AlreadyClosedException: this ReferenceManager is closed at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:97) at org.elasticsearch.index.engine.robin.RobinEngine.searcher(RobinEngine.java:739) at org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:1319) at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:516) at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:490) ... 8 more [2013-09-17 17:17:06,027][DEBUG][action.bulk ] [Eric] [test][2] failed to execute bulk item (index) index {[test][device][1156463], source[{"data................."}]} org.elasticsearch.index.engine.IndexFailedEngineException: [test][2] Index failed for [device#1156463] at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:503) at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:341) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:397) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:521) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.apache.lucene.store.AlreadyClosedException: this ReferenceManager is closed at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:97) at org.elasticsearch.index.engine.robin.RobinEngine.searcher(RobinEngine.java:739) at org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:1319) at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:516) at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:490) ... 8 more [2013-09-17 17:17:06,039][DEBUG][action.bulk ] [Eric] [test][2], node[vZI558XRT1i1kGPb4OXXnA], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.bulk.BulkShardRequest@5433050b] java.lang.NullPointerException [2013-09-17 17:17:06,038][DEBUG][action.bulk ] [Eric] [test][2], node[vZI558XRT1i1kGPb4OXXnA], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.bulk.BulkShardRequest@1fede67b] java.lang.NullPointerException [2013-09-17 17:17:06,044][WARN ][index.translog ] [Eric] [test][2] failed to flush shard on translog threshold org.elasticsearch.index.engine.FlushFailedEngineException: [test][2] Flush failed at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:895) at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:502) at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:186) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.lang.OutOfMemoryError: Java heap space [2013-09-17 17:17:06,045][DEBUG][action.bulk ] [Eric] [test][2] failed to execute bulk item (index) index {[test][device][1208327], source[{"data...."}]} org.elasticsearch.index.engine.IndexFailedEngineException: [test][2] Index failed for [device#1208327] at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:498) at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:341)

Any input much appreciated.

Thanks Pranav.

[starit1977]

Note - If, I increase the Heap size more than the default, then it works fine. But i don't know how much Heap size should be set ideally to make this work? The command that gave error used the Default heap size i.e. 256 MB. We are planning to use this plugin for backup/restore/export/import like 50GB of data.

Any input on the above error, much appreciated.

Thanks Pranav.

[mfussenegger]

This is probably related with #35 and should be fixed with 0.90.4

[starit1977]

I'm installing elastic search 0.90.5 version and see if i can get it working.

Between - i'm not a java guy, is there any simple set of steps that can help me install this plugin. I felt it is complicated, since again i'm not a java guy.

Thanks again for quick reply.

[apatrida]

for 0.90.5 see the pull request for that version, there are changes you need in the code to safely use all of the plugin