search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.42k stars 24.57k forks source link

Shard stuck in initializing state after OOM on a node #9086

Closed ppf2 closed 8 years ago

ppf2 commented 9 years ago

This may be related to https://github.com/elasticsearch/elasticsearch/issues/9018 . Pending_task output is not available for this incident and the end user has already restored from a snapshot to recover from the issue. But here are the sequence of events leading to the stuck shard in initializing state:

Original state:

image

The node ran oom when there were outstanding indexing and merge tasks:

[2014-12-23 19:07:36,687][INFO ][index.engine.internal    ] [ESD_dev_i-438764ae] [element_v1][0] stop throttling indexing: numMergesInFlight=3, maxNumMerges=4
[2014-12-23 19:09:01,196][INFO ][index.engine.internal    ] [ESD_dev_i-438764ae] [element_v1][9] stop throttling indexing: numMergesInFlight=3, maxNumMerges=4
[2014-12-23 19:13:27,845][INFO ][cluster.service          ] [ESD_dev_i-438764ae] removed {[SSA_dev_i-e38bf50f][fhi0ftB7TtSB2KUVNZYu7A][SSA_dev_i-e38bf50f][inet[/IP:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ESM_dev_i-febd3600][35xCsRzNTDmtqvcTJivZ1Q][ESM_dev_i-febd3600][inet[/IP:9300]]{aws_availability_zone=us-east-1d, data=false, master=true}])
[2014-12-23 19:15:40,285][INFO ][cluster.service          ] [ESD_dev_i-438764ae] added {[SSA_dev_i-e38bf50f][UzB7aWj1QSiaF6IUeKEXwg][SSA_dev_i-e38bf50f][inet[/IP:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[ESM_dev_i-febd3600][35xCsRzNTDmtqvcTJivZ1Q][ESM_dev_i-febd3600][inet[/IP:9300]]{aws_availability_zone=us-east-1d, data=false, master=true}])
[2014-12-23 19:25:41,147][ERROR][index.engine.internal    ] [ESD_dev_i-438764ae] [element_v1][0] Exception while waiting for merges asynchronously after optimize
org.elasticsearch.index.engine.FlushFailedEngineException: [element_v1][0] Flush failed
    at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:901)
    at org.elasticsearch.index.engine.internal.InternalEngine.waitForMerges(InternalEngine.java:1024)
    at org.elasticsearch.index.engine.internal.InternalEngine.access$200(InternalEngine.java:95)
    at org.elasticsearch.index.engine.internal.InternalEngine$2.run(InternalEngine.java:1072)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
    at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:698)
    at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:712)
    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3063)
    at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:891)
    ... 6 more
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:260)
    at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:213)
    at org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:176)
    at org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:586)
    at org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:248)
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:133)
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4173)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3768)
    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
    at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2014-12-23 19:25:41,340][DEBUG][action.admin.cluster.node.stats] [ESD_dev_i-438764ae] failed to execute on node [x9APfIW6Rii1WNPSElU0Pg]
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
    at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:698)
    at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:712)
    at org.apache.lucene.index.IndexWriter.ramBytesUsed(IndexWriter.java:462)
    at org.elasticsearch.index.engine.internal.InternalEngine.segmentsStats(InternalEngine.java:1193)
    at org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:555)
    at org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:170)
    at org.elasticsearch.action.admin.indices.stats.ShardStats.<init>(ShardStats.java:49)
    at org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
    at org.elasticsearch.node.service.NodeService.stats(NodeService.java:156)
    at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:96)
    at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:44)
    at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$2.run(TransportNodesOperationAction.java:141)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Not able to recover the shard [element_v1][9] given that it has been corrupted due to the OOM. You can see that the replica on the other node has been promoted to primary.

image

Caused by: org.elasticsearch.index.engine.EngineClosedException: [item_v1][7] CurrentState[CLOSED] 
    at org.elasticsearch.index.engine.internal.InternalEngine.ensureOpen(InternalEngine.java:963)
    at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:818)
    at org.elasticsearch.index.engine.internal.InternalEngine.snapshotIndex(InternalEngine.java:1086)
    at org.elasticsearch.index.shard.service.InternalIndexShard.snapshotIndex(InternalIndexShard.java:645)
    at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:84)
    ... 4 more
[2014-12-23 19:35:08,773][INFO ][node                     ] [ESD_dev_i-438764ae] version[1.4.2], pid[8608], build[927caff/2014-12-16T14:11:12Z]
[2014-12-23 19:35:08,773][INFO ][node                     ] [ESD_dev_i-438764ae] initializing ...
[2014-12-23 19:35:09,187][INFO ][plugins                  ] [ESD_dev_i-438764ae] loaded [analysis-kuromoji, marvel, analysis-stempel, cloud-aws, analysis-phonetic], sites [marvel]
[2014-12-23 19:35:15,899][INFO ][node                     ] [ESD_dev_i-438764ae] initialized
[2014-12-23 19:35:15,899][INFO ][node                     ] [ESD_dev_i-438764ae] starting ...
[2014-12-23 19:35:16,108][INFO ][transport                ] [ESD_dev_i-438764ae] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/IP:9300]}
[2014-12-23 19:35:16,208][INFO ][discovery                ] [ESD_dev_i-438764ae] cluster-es-dev/izYMnjBBT2io42-zN55aBQ
[2014-12-23 19:35:34,034][INFO ][cluster.routing.allocation.decider] [ESD_dev_i-438764ae] updating [cluster.routing.allocation.enable] from [ALL] to [NONE]
[2014-12-23 19:35:34,568][INFO ][http                     ] [ESD_dev_i-438764ae] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/IP:9200]}
[2014-12-23 19:35:34,569][INFO ][node                     ] [ESD_dev_i-438764ae] started
[2014-12-23 19:36:09,905][INFO ][cluster.routing.allocation.decider] [ESD_dev_i-438764ae] updating [cluster.routing.allocation.enable] from [NONE] to [ALL]
[2014-12-23 19:47:07,921][WARN ][indices.cluster          ] [ESD_dev_i-438764ae] [element_v1][9] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [element_v1][9] failed to fetch index version after copying it over
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:158)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [element_v1][9] shard allocated for local recovery (post api), should exist, but doesn't, current files: [recovery.1419363916358._3v4z_es090_0.tim, _3v1c.nvd, _3v1g_2.del, recovery.1419363916358._3uza_Lucene410_0.dvd, recovery.1419363916358._3v50_Lucene410_0.dvd, recovery.1419363916358._3v50_Lucene410_0.dvm, recovery.1419363916358._3v50_es090_0.tip, recovery.1419363916358._3v4n_es090_0.tim, recovery.1419363916358._3v50.fdx, recovery.1419363916358._3v4z_Lucene410_0.dvm, recovery.1419363916358._3v4z_es090_0.doc, _3v02.fdt, _3v0q_3.del, _3v4m.fnm, _3v4m.si, recovery.1419363916358._3v4r.fdx, segments_3u8, segments.gen, recovery.1419363916358._3uza.si, recovery.1419363916358._3uza_es090_0.tip, recovery.1419363916358._3uza_es090_0.doc, recovery.1419363916358._3v4p.nvd, recovery.1419363916358._3uza.fnm, recovery.1419363916358._3v4n.nvd, recovery.1419363916358._3v4p.fdt, recovery.1419363916358._3uza_es090_0.pos, recovery.1419363916358._3uza_Lucene410_0.dvm, recovery.1419363916358._3uza.fdt, recovery.1419363916358._3uza_es090_0.tim, recovery.1419363916358._3uza_es090_0.blm, recovery.1419363916358._3v50.fnm, recovery.1419363916358._3v50.fdt, recovery.1419363916358._3v50_es090_0.tim, recovery.1419363916358._3v50_es090_0.pos, recovery.1419363916358._3v4z_es090_0.pos, recovery.1419363916358._3v4z.nvm, recovery.1419363916358._3v4z.fnm, _3v02.nvd, _3v1g.fnm, recovery.1419363916358._3v4q_Lucene410_0.dvm, _3v4h.fdt, recovery.1419363916358._3v4z.nvd, recovery.1419363916358._3v4r.nvd, _3v1g.fdt, recovery.1419363916358._3v4z_es090_0.tip, _3v1c_es090_0.blm, recovery.1419363916358._3v4z.fdt, recovery.1419363916358._3v4z_es090_0.blm, _3v1g_es090_0.doc, _3v1c.si, _3v1g_es090_0.pos, _3v1c_Lucene410_0.dvm, _3v1g_es090_0.tim, _3v1c.fdt, _3v1g.fdx, recovery.1419363916358._3v4r.si, _3v1g_Lucene410_0.dvm, _3v1c_es090_0.pos, _3v1g.nvd, _3v1c_Lucene410_0.dvd, _3v1g.nvm, _3v1c_es090_0.tip, _3v1g_es090_0.tip, _3v1c.fnm, _3v0q.nvd, _3v4h.nvd, _3v1c_es090_0.doc, _3v0q_es090_0.tim, recovery.1419363916358._3v4z.fdx, recovery.1419363916358._3v4r.fdt, _3v1g_es090_0.blm, _3v02.fnm, _3v4o.fdx, _3v4l.cfs, _3v02.fdx, _3v1g_Lucene410_0.dvd, _3v02_Lucene410_0.dvm, _3v0q.fdt, _3v1c.fdx, _3v4h.fdx, recovery.1419363916358._3v4r.fnm, _3v0q.fdx, _3v1c.nvm, _3v4l.cfe, _3uzk_1.del, _3v02_es090_0.tim, _3v0q_es090_0.doc, _3v02_es090_0.doc, _3v0q_es090_0.pos, recovery.1419363916358._3uza.nvd, recovery.1419363916358._3uza.fdx, recovery.1419363916358._3uza.nvm, recovery.1419363916358._3v50_es090_0.blm, recovery.1419363916358._3v50.nvm, recovery.1419363916358._3v50.si, recovery.1419363916358._3v50.nvd, recovery.1419363916358._3v50_es090_0.doc, recovery.1419363916358._3v4z.si, recovery.1419363916358._3v4z_Lucene410_0.dvd, _3v1c_es090_0.tim, recovery.1419363916358._3v4t_es090_0.tim, _3v4h.fnm, _3v4o.nvd, recovery.1419363916358._3v4n_Lucene410_0.dvd, recovery.1419363916358._3v4t.nvm, _3v4h_es090_0.pos, recovery.1419363916358._3v4t.fdx, _3v4o.nvm, _3uz0.fdx, _3v4h_Lucene410_0.dvm, _3uz0.nvd, _3v4h_es090_0.tim, _3uz0.fdt, _3v4h_es090_0.doc, _3v02_es090_0.tip, _3v4q_Lucene410_0.dvm, recovery.1419363916358._3v4q_Lucene410_0.dvd, _3uz0_Lucene410_0.dvm, _3v4o.fdt, recovery.1419363916358._3v4n_es090_0.blm, _3v4o_es090_0.doc, _3uz0.fnm, _3v4o_es090_0.pos, _3v02_es090_0.blm, _3v4l.si, recovery.1419363916358._3v4t.si, _3v4o_es090_0.tim, _3v02_es090_0.pos, _3v4h.nvm, _3v02.si, _3v4o_es090_0.tip, recovery.1419363916358._3v4n.fdx, recovery.1419363916358._3v4t.nvd, recovery.1419363916358._3v4n_es090_0.tip, _3v4h_es090_0.tip, _3uz0.si, _3v4q_Lucene410_0.dvd, recovery.1419363916358._3v4n_Lucene410_0.dvm, _3v02_Lucene410_0.dvd, _3v4h_es090_0.blm, _3v02.nvm, _3v4h_Lucene410_0.dvd, recovery.1419363916358._3v4n.fnm, recovery.1419363916358._3v4t.fnm, _17sq_es090_0.tip, _3v4h.cfs, _17sq_es090_0.tim, _3v4h.cfe, _17sq.nvd, _3uz0_es090_0.tip, _17sq_es090_0.blm, _3uz0_es090_0.tim, _17sq.nvm, _3uz0_es090_0.doc, _17sq_es090_0.doc, _3uza.fdx, _17sq.si, _3uzk.fdx, _17sq.fdx, _3v0q_es090_0.tip, _17sq.fdt, _3uza_1.del, _17sq.fnm, _3uza.fdt, _17sq_Lucene45_0.dvd, _3v0q_es090_0.blm, _17sq_Lucene45_0.dvm, _3uz0_Lucene410_0.dvd, _17sq_es090_0.pos, recovery.1419363916358._3v4r_es090_0.tim, recovery.1419363916358._3v4r_es090_0.doc, recovery.1419363916358._3v4r_es090_0.tip, recovery.1419363916358._3v4r_es090_0.pos, recovery.1419363916358._3v4r_es090_0.blm, recovery.1419363916358._3v4r_Lucene410_0.dvd, recovery.1419363916358._3v4r.nvm, recovery.1419363916358._3v4s.fdx, recovery.1419363916358._3v4s_es090_0.tip, recovery.1419363916358._3v4s_es090_0.blm, recovery.1419363916358._3v4s_Lucene410_0.dvm, recovery.1419363916358._3v4s_es090_0.doc, recovery.1419363916358._3v4s_Lucene410_0.dvd, recovery.1419363916358._3v4t.fdt, recovery.1419363916358._3v4t_es090_0.pos, _3v0q_Lucene410_0.dvd, _3uz0_es090_0.blm, _3uzk.fdt, recovery.1419363916358._3v4u.nvd, _3uzk_es090_0.doc, recovery.1419363916358._3v4n_es090_0.doc, recovery.1419363916358._3v4r_Lucene410_0.dvm, recovery.1419363916358._3v4s.fnm, recovery.1419363916358._3v4s.fdt, recovery.1419363916358._3v4s_es090_0.tim, recovery.1419363916358._3v4s.nvd, recovery.1419363916358._3v4s.si, recovery.1419363916358._3v4s.nvm, recovery.1419363916358._3v4s_es090_0.pos, recovery.1419363916358._3v4t_Lucene410_0.dvd, recovery.1419363916358._3v4v.si, _3v4i.cfs, _3v4j.fnm, _3v4i.cfe, recovery.1419363916358._3v4u_es090_0.doc, recovery.1419363916358._3v4v.nvm, _3uzk_Lucene410_0.dvm, _3uz0_es090_0.pos, _checksums-1419291672870, _3v4l_es090_0.tip, _3v4o_Lucene410_0.dvd, _3v4l.fnm, _3uzk.nvd, _3uz0.nvm, _3uzk.nvm, _3v4f.si, _3uzk.fnm, recovery.1419363916358._3v4t_es090_0.tip, recovery.1419363916358._3v4t_es090_0.blm, recovery.1419363916358._3v4t_es090_0.doc, recovery.1419363916358._3v4u_Lucene410_0.dvd, _3v4l_Lucene410_0.dvm, _17sq_27.del, recovery.1419363916358._3v4u_es090_0.tim, recovery.1419363916358._3v4u.fdt, _3v0q.nvm, recovery.1419363916358._3v4u_es090_0.tip, recovery.1419363916358._3v4u.si, _3v4o_Lucene410_0.dvm, _3v4f.cfs, _3v0q_Lucene410_0.dvm, recovery.1419363916358._3v4t_Lucene410_0.dvm, recovery.1419363916358._3v4u_Lucene410_0.dvm, recovery.1419363916358._3v4u.fdx, write.lock, recovery.1419363916358._3v4v.fdx, recovery.1419363916358._3v4n.si, _3v4r.fdx, recovery.1419363916358._3v4v.nvd, _3v4r.fdt, recovery.1419363916358._3v4v.fnm, _3v4r_es090_0.doc, _3v4l_es090_0.blm, _3v4r_es090_0.pos, _3v4l.nvm, _3v4r_es090_0.tim, _3v4e.nvm, _3uzk.si, recovery.1419363916358._3v4v.fdt, _3v4r_es090_0.tip, _3v4f.cfe, _3uza_es090_0.pos, _3v4e_es090_0.tip, _3uza.fnm, _3v4e.nvd, _3uza.si, _3v4e_es090_0.tim, _3uza_es090_0.tim, _3v4e_Lucene410_0.dvd, _3v4r_es090_0.blm, recovery.1419363916358._3v4u.nvm, recovery.1419363916358._3v4u_es090_0.pos, _3uza_es090_0.tip, _3v4l_Lucene410_0.dvd, _3v4r_Lucene410_0.dvd, _3v4j_es090_0.tip, _3v4o_es090_0.blm, _3v4e_es090_0.doc, _3v4h_1.del, recovery.1419363916358._3v4w.nvd, _3v4r_Lucene410_0.dvm, _3v4j.si, _3v4h.si, _3v4e.fnm, _3v4r.nvd, _3v4e_Lucene410_0.dvm, _3v1g.si, recovery.1419363916358._3v4w.fdx, _3v4r.nvm, recovery.1419363916358._3v4w.nvm, _3uzk_es090_0.pos, _3uzk_es090_0.tim, _3uzk_es090_0.tip, _3uzk_es090_0.blm, _3v4i.si, _3uza_es090_0.doc, _3uzk_Lucene410_0.dvd, recovery.1419363916358._3v4w.fnm, _3v4r.fnm, recovery.1419363916358._3v4w.fdt, _3v4r.si, _3v4e.fdt, _3v4s.fdx, recovery.1419363916358._3v4w.si, _3v4s.fdt, recovery.1419363916358._3v4x.fdx, _3v4s_es090_0.doc, _3v4j_Lucene410_0.dvm, _3v4s_es090_0.pos, _3v4e_es090_0.blm, _3v0q.fnm, _3v4e.fdx, _3v4s_es090_0.tim, _3v4e.si, _3v4s_es090_0.tip, recovery.1419363916358._3v4u.fnm, _3v4e_es090_0.pos, _3v4s.nvd, _3v0q.si, _3v4l.fdx, _3v4s.nvm, _3v52.si, _3v4k.cfs, _3v4k.cfe, _3v4k.si, _3v4s_Lucene410_0.dvd, _3v4s.fnm, _3v4s.si, _3v4t.fdt, _3v4t_es090_0.pos, _3v4t_es090_0.tip, _3v4t_Lucene410_0.dvd, _3v4t.nvd, _3v4t.fnm, _3v4u.fdx, _3v4u_es090_0.doc, _3v4u_es090_0.tim, _3v4u_es090_0.blm, _3v4u_Lucene410_0.dvm, _3v4u.nvm, _3v4u.si, _3v4v.fdt, _3v4v_es090_0.pos, _3v4v_es090_0.tip, _3v4v_Lucene410_0.dvd, _3v4v.nvd, _3v4v.fnm, _3v4w.fdx, _3v4w_es090_0.doc, _3v4w_es090_0.tim, _3v4w_es090_0.blm, _3v4w_Lucene410_0.dvm, _3v4w.nvm, _3v4w.si, _3v4x.fdt, _3v4x_es090_0.pos, _3v4x_es090_0.tip, _3v4x_Lucene410_0.dvd, _3v4x.nvd, _3v4x.fnm, _3v4y.fdx, _3v4y_es090_0.doc, _3v4y_es090_0.tim, _3v4y_es090_0.blm, _3v4y_Lucene410_0.dvm, _3v4y.nvm, _3v4y.si, _3v4z.fdt, _3v4z_es090_0.pos, _3v4z_es090_0.tip, _3v4z_Lucene410_0.dvd, _3v4z.nvd, _3v4z.fnm, _3v50.fdx, _3v50_es090_0.doc, _3v50_es090_0.tim, _3v50_es090_0.blm, _3v50_Lucene410_0.dvm, _3v50.nvm, _3v50.si, _3v51.fdt, _3v51_es090_0.pos, _3v51_es090_0.tip, _3v51_Lucene410_0.dvd, _3v51.nvd, _3v51.fnm, _3v52.fdx, _3v52_es090_0.doc, _3v52_es090_0.tim, _3v52_es090_0.blm, _3v52_Lucene410_0.dvm, _3v52.nvm, _3v53.fdx, _3uza_es090_0.blm, _3uza_Lucene410_0.dvd, _3uza_Lucene410_0.dvm, _3uza.nvd, _3uza.nvm, _3v4s_es090_0.blm, _3v4s_Lucene410_0.dvm, _3v4t.fdx, _3v4t_es090_0.doc, _3v4t_es090_0.tim, _3v4t_es090_0.blm, _3v4t_Lucene410_0.dvm, _3v4t.nvm, _3v4t.si, _3v4u.fdt, _3v4u_es090_0.pos, _3v4u_es090_0.tip, _3v4u_Lucene410_0.dvd, _3v4u.nvd, _3v4u.fnm, _3v4v.fdx, _3v4v_es090_0.doc, _3v4v_es090_0.tim, _3v4v_es090_0.blm, _3v4v_Lucene410_0.dvm, _3v4v.nvm, _3v4v.si, _3v4w.fdt, _3v4w_es090_0.pos, _3v4w_es090_0.tip, _3v4w_Lucene410_0.dvd, _3v4w.nvd, _3v4w.fnm, _3v4x.fdx, _3v4x_es090_0.doc, _3v4x_es090_0.tim, _3v4x_es090_0.blm, _3v4x_Lucene410_0.dvm, _3v4x.nvm, _3v4x.si, _3v4y.fdt, _3v4y_es090_0.pos, _3v4y_es090_0.tip, _3v4y_Lucene410_0.dvd, _3v4y.nvd, _3v4y.fnm, _3v4z.fdx, _3v4z_es090_0.doc, _3v4z_es090_0.tim, _3v4z_es090_0.blm, _3v4z_Lucene410_0.dvm, _3v4z.nvm, _3v4z.si, _3v50.fdt, _3v50_es090_0.pos, _3v50_es090_0.tip, _3v50_Lucene410_0.dvd, _3v50.nvd, _3v50.fnm, _3v51.fdx, _3v51_es090_0.doc, _3v51_es090_0.tim, _3v51_es090_0.blm, _3v51_Lucene410_0.dvm, _3v51.nvm, _3v51.si, _3v52.fdt, _3v52_es090_0.pos, _3v52_es090_0.tip, _3v52_Lucene410_0.dvd, _3v52.nvd, _3v52.fnm, _3v53.fdt, _3v53_es090_0.doc, _3v53_es090_0.pos, _3v53_es090_0.tim, _3v53_es090_0.tip, _3v53_es090_0.blm, _3v53_Lucene410_0.dvd, _3v53_Lucene410_0.dvm, _3v53.nvd, _3v53.nvm, _3v53.fnm, _3v53.si, _3v4p.fdx, _3v4p.fdt, _3v4o.fnm, _3v4o.si, _3v4n.fdx, _3v4n.fdt, _3v4p_es090_0.doc, _3v4p_es090_0.pos, _3v4p_es090_0.tim, _3v4p_es090_0.tip, _3v4n_es090_0.doc, _3v4n_es090_0.pos, _3v4n_es090_0.tim, _3v4n_es090_0.tip, _3v4p_es090_0.blm, _3v4p_Lucene410_0.dvd, _3v4p_Lucene410_0.dvm, _3v4p.nvd, _3v4p.nvm, _3v4n_es090_0.blm, _3v4n_Lucene410_0.dvd, _3v4n_Lucene410_0.dvm, _3v4n.nvd, _3v4n.nvm, _3v4n.fnm, _3v4n.si, _3v4m.fdx, _3v4m.fdt, _3v4p.fnm, _3v4p.si, _3v4m_es090_0.doc, _3v4m_es090_0.pos, _3v4m_es090_0.tim, _3v4m_es090_0.tip, _3v4m_es090_0.blm, _3v4m_Lucene410_0.dvd, _3v4m_Lucene410_0.dvm, _3v4m.nvd, _3v4m.nvm, segments_3u6, recovery.1419363916358._3v4u_es090_0.blm, recovery.1419363916358._3v4v_es090_0.pos, recovery.1419363916358._3v4v_Lucene410_0.dvd, recovery.1419363916358._3v4v_Lucene410_0.dvm, recovery.1419363916358._3v4v_es090_0.tip, recovery.1419363916358._3v4v_es090_0.blm, recovery.1419363916358._3v4v_es090_0.doc, recovery.1419363916358._3v4v_es090_0.tim, recovery.1419363916358._3v4w_es090_0.doc, recovery.1419363916358._3v4w_Lucene410_0.dvm, recovery.1419363916358._3v4w_es090_0.pos, recovery.1419363916358._3v4w_Lucene410_0.dvd, recovery.1419363916358._3v4w_es090_0.blm, recovery.1419363916358._3v4w_es090_0.tim, recovery.1419363916358._3v4w_es090_0.tip, recovery.1419363916358._3v4x_Lucene410_0.dvd, recovery.1419363916358._3v4x_es090_0.tim, recovery.1419363916358._3v4x_es090_0.pos, recovery.1419363916358._3v4x_es090_0.tip, recovery.1419363916358._3v4x_es090_0.doc, recovery.1419363916358._3v4x_es090_0.blm, recovery.1419363916358._3v4x.si, recovery.1419363916358._3v4x.nvm, recovery.1419363916358._3v4x.fnm, recovery.1419363916358._3v4x.fdt, recovery.1419363916358._3v4x_Lucene410_0.dvm, recovery.1419363916358._3v4y.fdx, recovery.1419363916358._3v4x.nvd, recovery.1419363916358._3v4y_Lucene410_0.dvm, recovery.1419363916358._3v4y.nvd, recovery.1419363916358._3v4y_es090_0.tim, recovery.1419363916358._3v4y_es090_0.tip, recovery.1419363916358._3v4y_es090_0.blm, recovery.1419363916358._3v4y.nvm, recovery.1419363916358._3v4y_es090_0.doc, recovery.1419363916358._3v4y_es090_0.pos, recovery.1419363916358._3v4y.si, recovery.1419363916358._3v4y_Lucene410_0.dvd, recovery.1419363916358._3v4y.fnm, recovery.1419363916358._3v4y.fdt, recovery.1419363916358._3v4h.si, recovery.1419363916358._3v4i.cfs, recovery.1419363916358._3v4i.cfe, recovery.1419363916358._3v4i.si, recovery.1419363916358._3v4p_Lucene410_0.dvm, recovery.1419363916358._3v4p.fnm, recovery.1419363916358._3v4p.nvm, recovery.1419363916358._3v4p_es090_0.blm, recovery.1419363916358._3v4p_Lucene410_0.dvd, recovery.1419363916358._3v4p_es090_0.tim, recovery.1419363916358._3v4p_es090_0.tip, recovery.1419363916358._3v4p.si, recovery.1419363916358._3v4p.fdx, recovery.1419363916358._3v4p_es090_0.doc, recovery.1419363916358._3v4p_es090_0.pos, recovery.1419363916358._3v4n.nvm, recovery.1419363916358._3v4o_Lucene410_0.dvm, recovery.1419363916358._3v4o_es090_0.tip, recovery.1419363916358._3v4o.nvm, recovery.1419363916358._3v4o_es090_0.doc, recovery.1419363916358._3v4o_es090_0.pos, recovery.1419363916358._3v4o_es090_0.tim, recovery.1419363916358._3v4o_Lucene410_0.dvd, recovery.1419363916358._3v4o.fnm, recovery.1419363916358._3v4o.si, recovery.1419363916358._3v4o.fdt, recovery.1419363916358._3v4o.nvd, recovery.1419363916358._3v4o_es090_0.blm, recovery.1419363916358._3v4o.fdx, recovery.1419363916358._3v4l.si, recovery.1419363916358._3v4m_Lucene410_0.dvm, recovery.1419363916358._3v4m_Lucene410_0.dvd, recovery.1419363916358._3v4m.fdx, recovery.1419363916358._3v4m.si, recovery.1419363916358._3v4m.fnm, recovery.1419363916358._3v4m.nvm, recovery.1419363916358._3v4m_es090_0.tip, recovery.1419363916358._3v4m_es090_0.blm, recovery.1419363916358._3v4k.si, recovery.1419363916358._3v4n.fdt, _3v4j_es090_0.blm, _3v4j.fdx, _3v4j_es090_0.pos, _3v4j_Lucene410_0.dvd, _3v4j.nvm, _3v4k.fnm, _3v4k_Lucene410_0.dvd, _3v4k.fdx, _3v4k_es090_0.tip, _3v4k_Lucene410_0.dvm, _3v4k_es090_0.blm, _3v4k.nvm, _3v4e_1.del, _3v02_2.del, _3v1c_4.del, _3uz0_1.del, segments_3sh, recovery.1419363916358._3v4n_es090_0.pos, recovery.1419363916358._3v4m.nvd, recovery.1419363916358._3v4m_es090_0.doc, recovery.1419363916358._3v4m_es090_0.pos, recovery.1419363916358._3v4m.fdt, recovery.1419363916358._3v4m_es090_0.tim, _3v4l.nvd, _3v4l_es090_0.pos, _3v4l.fdt, _3v4l_es090_0.doc, _3v4l_es090_0.tim, _3v4j_es090_0.doc, _3v4j.fdt, _3v4j.nvd, _3v4j_es090_0.tim, _3v4k.nvd, _3v4k.fdt, _3v4k_es090_0.pos, _3v4k_es090_0.doc, _3v4k_es090_0.tim]
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:132)
    ... 4 more
Caused by: java.io.FileNotFoundException: No such file [_3v4q.si]
    at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
    at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
    at org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
    at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:487)
    at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
    at org.apache.lucene.codecs.lucene46.Lucene46SegmentInfoReader.read(Lucene46SegmentInfoReader.java:51)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:358)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:454)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:906)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:85)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:123)
    ... 4 more

It then appears to do a rebalancing and tried to initialize the shard again on the original node. This is when the shard gets stuck in initializing state on this node.

image

Could it be that we didn't clean up the corrupted files on disk when the shard data on the original node got corrupted due to the OOM, and when it tried to rebalance the shard back to this original node, it ended up using some of the corrupted data files on disk? 

[2014-12-23 19:47:07,942][WARN ][cluster.action.shard     ] [ESD_dev_i-438764ae] [element_v1][9] sending failed shard for [element_v1][9], node[izYMnjBBT2io42-zN55aBQ], [P], s[INITIALIZING], indexUUID [BCInnRQRSGK4c8B22qEB0g], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[element_v1][9] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[element_v1][9] shard allocated for local recovery (post api), should exist, but doesn't, current files: [recovery.1419363916358._3v4z_es090_0.tim, _3v1c.nvd, _3v1g_2.del, recovery.1419363916358._3uza_Lucene410_0.dvd, recovery.1419363916358._3v50_Lucene410_0.dvd, recovery.1419363916358._3v50_Lucene410_0.dvm, recovery.1419363916358._3v50_es090_0.tip, recovery.1419363916358._3v4n_es090_0.tim, recovery.1419363916358._3v50.fdx, recovery.1419363916358._3v4z_Lucene410_0.dvm, recovery.1419363916358._3v4z_es090_0.doc, _3v02.fdt, _3v0q_3.del, _3v4m.fnm, _3v4m.si, recovery.1419363916358._3v4r.fdx, segments_3u8, segments.gen, recovery.1419363916358._3uza.si, recovery.1419363916358._3uza_es090_0.tip, recovery.1419363916358._3uza_es090_0.doc, recovery.1419363916358._3v4p.nvd, recovery.1419363916358._3uza.fnm, recovery.1419363916358._3v4n.nvd, recovery.1419363916358._3v4p.fdt, recovery.1419363916358._3uza_es090_0.pos, recovery.1419363916358._3uza_Lucene410_0.dvm, recovery.1419363916358._3uza.fdt, recovery.1419363916358._3uza_es090_0.tim, recovery.1419363916358._3uza_es090_0.blm, recovery.1419363916358._3v50.fnm, recovery.1419363916358._3v50.fdt, recovery.1419363916358._3v50_es090_0.tim, recovery.1419363916358._3v50_es090_0.pos, recovery.1419363916358._3v4z_es090_0.pos, recovery.1419363916358._3v4z.nvm, recovery.1419363916358._3v4z.fnm, _3v02.nvd, _3v1g.fnm, recovery.1419363916358._3v4q_Lucene410_0.dvm, _3v4h.fdt, recovery.1419363916358._3v4z.nvd, recovery.1419363916358._3v4r.nvd, _3v1g.fdt, recovery.1419363916358._3v4z_es090_0.tip, _3v1c_es090_0.blm, recovery.1419363916358._3v4z.fdt, recovery.1419363916358._3v4z_es090_0.blm, _3v1g_es090_0.doc, _3v1c.si, _3v1g_es090_0.pos, _3v1c_Lucene410_0.dvm, _3v1g_es090_0.tim, _3v1c.fdt, _3v1g.fdx, recovery.1419363916358._3v4r.si, _3v1g_Lucene410_0.dvm, _3v1c_es090_0.pos, _3v1g.nvd, _3v1c_Lucene410_0.dvd, _3v1g.nvm, _3v1c_es090_0.tip, _3v1g_es090_0.tip, _3v1c.fnm, _3v0q.nvd, _3v4h.nvd, _3v1c_es090_0.doc, _3v0q_es090_0.tim, recovery.1419363916358._3v4z.fdx, recovery.1419363916358._3v4r.fdt, _3v1g_es090_0.blm, _3v02.fnm, _3v4o.fdx, _3v4l.cfs, _3v02.fdx, _3v1g_Lucene410_0.dvd, _3v02_Lucene410_0.dvm, _3v0q.fdt, _3v1c.fdx, _3v4h.fdx, recovery.1419363916358._3v4r.fnm, _3v0q.fdx, _3v1c.nvm, _3v4l.cfe, _3uzk_1.del, _3v02_es090_0.tim, _3v0q_es090_0.doc, _3v02_es090_0.doc, _3v0q_es090_0.pos, recovery.1419363916358._3uza.nvd, recovery.1419363916358._3uza.fdx, recovery.1419363916358._3uza.nvm, recovery.1419363916358._3v50_es090_0.blm, recovery.1419363916358._3v50.nvm, recovery.1419363916358._3v50.si, recovery.1419363916358._3v50.nvd, recovery.1419363916358._3v50_es090_0.doc, recovery.1419363916358._3v4z.si, recovery.1419363916358._3v4z_Lucene410_0.dvd, _3v1c_es090_0.tim, recovery.1419363916358._3v4t_es090_0.tim, _3v4h.fnm, _3v4o.nvd, recovery.1419363916358._3v4n_Lucene410_0.dvd, recovery.1419363916358._3v4t.nvm, _3v4h_es090_0.pos, recovery.1419363916358._3v4t.fdx, _3v4o.nvm, _3uz0.fdx, _3v4h_Lucene410_0.dvm, _3uz0.nvd, _3v4h_es090_0.tim, _3uz0.fdt, _3v4h_es090_0.doc, _3v02_es090_0.tip, _3v4q_Lucene410_0.dvm, recovery.1419363916358._3v4q_Lucene410_0.dvd, _3uz0_Lucene410_0.dvm, _3v4o.fdt, recovery.1419363916358._3v4n_es090_0.blm, _3v4o_es090_0.doc, _3uz0.fnm, _3v4o_es090_0.pos, _3v02_es090_0.blm, _3v4l.si, recovery.1419363916358._3v4t.si, _3v4o_es090_0.tim, _3v02_es090_0.pos, _3v4h.nvm, _3v02.si, _3v4o_es090_0.tip, recovery.1419363916358._3v4n.fdx, recovery.1419363916358._3v4t.nvd, recovery.1419363916358._3v4n_es090_0.tip, _3v4h_es090_0.tip, _3uz0.si, _3v4q_Lucene410_0.dvd, recovery.1419363916358._3v4n_Lucene410_0.dvm, _3v02_Lucene410_0.dvd, _3v4h_es090_0.blm, _3v02.nvm, _3v4h_Lucene410_0.dvd, recovery.1419363916358._3v4n.fnm, recovery.1419363916358._3v4t.fnm, _17sq_es090_0.tip, _3v4h.cfs, _17sq_es090_0.tim, _3v4h.cfe, _17sq.nvd, _3uz0_es090_0.tip, _17sq_es090_0.blm, _3uz0_es090_0.tim, _17sq.nvm, _3uz0_es090_0.doc, _17sq_es090_0.doc, _3uza.fdx, _17sq.si, _3uzk.fdx, _17sq.fdx, _3v0q_es090_0.tip, _17sq.fdt, _3uza_1.del, _17sq.fnm, _3uza.fdt, _17sq_Lucene45_0.dvd, _3v0q_es090_0.blm, _17sq_Lucene45_0.dvm, _3uz0_Lucene410_0.dvd, _17sq_es090_0.pos, recovery.1419363916358._3v4r_es090_0.tim, recovery.1419363916358._3v4r_es090_0.doc, recovery.1419363916358._3v4r_es090_0.tip, recovery.1419363916358._3v4r_es090_0.pos, recovery.1419363916358._3v4r_es090_0.blm, recovery.1419363916358._3v4r_Lucene410_0.dvd, recovery.1419363916358._3v4r.nvm, recovery.1419363916358._3v4s.fdx, recovery.1419363916358._3v4s_es090_0.tip, recovery.1419363916358._3v4s_es090_0.blm, recovery.1419363916358._3v4s_Lucene410_0.dvm, recovery.1419363916358._3v4s_es090_0.doc, recovery.1419363916358._3v4s_Lucene410_0.dvd, recovery.1419363916358._3v4t.fdt, recovery.1419363916358._3v4t_es090_0.pos, _3v0q_Lucene410_0.dvd, _3uz0_es090_0.blm, _3uzk.fdt, recovery.1419363916358._3v4u.nvd, _3uzk_es090_0.doc, recovery.1419363916358._3v4n_es090_0.doc, recovery.1419363916358._3v4r_Lucene410_0.dvm, recovery.1419363916358._3v4s.fnm, recovery.1419363916358._3v4s.fdt, recovery.1419363916358._3v4s_es090_0.tim, recovery.1419363916358._3v4s.nvd, recovery.1419363916358._3v4s.si, recovery.1419363916358._3v4s.nvm, recovery.1419363916358._3v4s_es090_0.pos, recovery.1419363916358._3v4t_Lucene410_0.dvd, recovery.1419363916358._3v4v.si, _3v4i.cfs, _3v4j.fnm, _3v4i.cfe, recovery.1419363916358._3v4u_es090_0.doc, recovery.1419363916358._3v4v.nvm, _3uzk_Lucene410_0.dvm, _3uz0_es090_0.pos, _checksums-1419291672870, _3v4l_es090_0.tip, _3v4o_Lucene410_0.dvd, _3v4l.fnm, _3uzk.nvd, _3uz0.nvm, _3uzk.nvm, _3v4f.si, _3uzk.fnm, recovery.1419363916358._3v4t_es090_0.tip, recovery.1419363916358._3v4t_es090_0.blm, recovery.1419363916358._3v4t_es090_0.doc, recovery.1419363916358._3v4u_Lucene410_0.dvd, _3v4l_Lucene410_0.dvm, _17sq_27.del, recovery.1419363916358._3v4u_es090_0.tim, recovery.1419363916358._3v4u.fdt, _3v0q.nvm, recovery.1419363916358._3v4u_es090_0.tip, recovery.1419363916358._3v4u.si, _3v4o_Lucene410_0.dvm, _3v4f.cfs, _3v0q_Lucene410_0.dvm, recovery.1419363916358._3v4t_Lucene410_0.dvm, recovery.1419363916358._3v4u_Lucene410_0.dvm, recovery.1419363916358._3v4u.fdx, write.lock, recovery.1419363916358._3v4v.fdx, recovery.1419363916358._3v4n.si, _3v4r.fdx, recovery.1419363916358._3v4v.nvd, _3v4r.fdt, recovery.1419363916358._3v4v.fnm, _3v4r_es090_0.doc, _3v4l_es090_0.blm, _3v4r_es090_0.pos, _3v4l.nvm, _3v4r_es090_0.tim, _3v4e.nvm, _3uzk.si, recovery.1419363916358._3v4v.fdt, _3v4r_es090_0.tip, _3v4f.cfe, _3uza_es090_0.pos, _3v4e_es090_0.tip, _3uza.fnm, _3v4e.nvd, _3uza.si, _3v4e_es090_0.tim, _3uza_es090_0.tim, _3v4e_Lucene410_0.dvd, _3v4r_es090_0.blm, recovery.1419363916358._3v4u.nvm, recovery.1419363916358._3v4u_es090_0.pos, _3uza_es090_0.tip, _3v4l_Lucene410_0.dvd, _3v4r_Lucene410_0.dvd, _3v4j_es090_0.tip, _3v4o_es090_0.blm, _3v4e_es090_0.doc, _3v4h_1.del, recovery.1419363916358._3v4w.nvd, _3v4r_Lucene410_0.dvm, _3v4j.si, _3v4h.si, _3v4e.fnm, _3v4r.nvd, _3v4e_Lucene410_0.dvm, _3v1g.si, recovery.1419363916358._3v4w.fdx, _3v4r.nvm, recovery.1419363916358._3v4w.nvm, _3uzk_es090_0.pos, _3uzk_es090_0.tim, _3uzk_es090_0.tip, _3uzk_es090_0.blm, _3v4i.si, _3uza_es090_0.doc, _3uzk_Lucene410_0.dvd, recovery.1419363916358._3v4w.fnm, _3v4r.fnm, recovery.1419363916358._3v4w.fdt, _3v4r.si, _3v4e.fdt, _3v4s.fdx, recovery.1419363916358._3v4w.si, _3v4s.fdt, recovery.1419363916358._3v4x.fdx, _3v4s_es090_0.doc, _3v4j_Lucene410_0.dvm, _3v4s_es090_0.pos, _3v4e_es090_0.blm, _3v0q.fnm, _3v4e.fdx, _3v4s_es090_0.tim, _3v4e.si, _3v4s_es090_0.tip, recovery.1419363916358._3v4u.fnm, _3v4e_es090_0.pos, _3v4s.nvd, _3v0q.si, _3v4l.fdx, _3v4s.nvm, _3v52.si, _3v4k.cfs, _3v4k.cfe, _3v4k.si, _3v4s_Lucene410_0.dvd, _3v4s.fnm, _3v4s.si, _3v4t.fdt, _3v4t_es090_0.pos, _3v4t_es090_0.tip, _3v4t_Lucene410_0.dvd, _3v4t.nvd, _3v4t.fnm, _3v4u.fdx, _3v4u_es090_0.doc, _3v4u_es090_0.tim, _3v4u_es090_0.blm, _3v4u_Lucene410_0.dvm, _3v4u.nvm, _3v4u.si, _3v4v.fdt, _3v4v_es090_0.pos, _3v4v_es090_0.tip, _3v4v_Lucene410_0.dvd, _3v4v.nvd, _3v4v.fnm, _3v4w.fdx, _3v4w_es090_0.doc, _3v4w_es090_0.tim, _3v4w_es090_0.blm, _3v4w_Lucene410_0.dvm, _3v4w.nvm, _3v4w.si, _3v4x.fdt, _3v4x_es090_0.pos, _3v4x_es090_0.tip, _3v4x_Lucene410_0.dvd, _3v4x.nvd, _3v4x.fnm, _3v4y.fdx, _3v4y_es090_0.doc, _3v4y_es090_0.tim, _3v4y_es090_0.blm, _3v4y_Lucene410_0.dvm, _3v4y.nvm, _3v4y.si, _3v4z.fdt, _3v4z_es090_0.pos, _3v4z_es090_0.tip, _3v4z_Lucene410_0.dvd, _3v4z.nvd, _3v4z.fnm, _3v50.fdx, _3v50_es090_0.doc, _3v50_es090_0.tim, _3v50_es090_0.blm, _3v50_Lucene410_0.dvm, _3v50.nvm, _3v50.si, _3v51.fdt, _3v51_es090_0.pos, _3v51_es090_0.tip, _3v51_Lucene410_0.dvd, _3v51.nvd, _3v51.fnm, _3v52.fdx, _3v52_es090_0.doc, _3v52_es090_0.tim, _3v52_es090_0.blm, _3v52_Lucene410_0.dvm, _3v52.nvm, _3v53.fdx, _3uza_es090_0.blm, _3uza_Lucene410_0.dvd, _3uza_Lucene410_0.dvm, _3uza.nvd, _3uza.nvm, _3v4s_es090_0.blm, _3v4s_Lucene410_0.dvm, _3v4t.fdx, _3v4t_es090_0.doc, _3v4t_es090_0.tim, _3v4t_es090_0.blm, _3v4t_Lucene410_0.dvm, _3v4t.nvm, _3v4t.si, _3v4u.fdt, _3v4u_es090_0.pos, _3v4u_es090_0.tip, _3v4u_Lucene410_0.dvd, _3v4u.nvd, _3v4u.fnm, _3v4v.fdx, _3v4v_es090_0.doc, _3v4v_es090_0.tim, _3v4v_es090_0.blm, _3v4v_Lucene410_0.dvm, _3v4v.nvm, _3v4v.si, _3v4w.fdt, _3v4w_es090_0.pos, _3v4w_es090_0.tip, _3v4w_Lucene410_0.dvd, _3v4w.nvd, _3v4w.fnm, _3v4x.fdx, _3v4x_es090_0.doc, _3v4x_es090_0.tim, _3v4x_es090_0.blm, _3v4x_Lucene410_0.dvm, _3v4x.nvm, _3v4x.si, _3v4y.fdt, _3v4y_es090_0.pos, _3v4y_es090_0.tip, _3v4y_Lucene410_0.dvd, _3v4y.nvd, _3v4y.fnm, _3v4z.fdx, _3v4z_es090_0.doc, _3v4z_es090_0.tim, _3v4z_es090_0.blm, _3v4z_Lucene410_0.dvm, _3v4z.nvm, _3v4z.si, _3v50.fdt, _3v50_es090_0.pos, _3v50_es090_0.tip, _3v50_Lucene410_0.dvd, _3v50.nvd, _3v50.fnm, _3v51.fdx, _3v51_es090_0.doc, _3v51_es090_0.tim, _3v51_es090_0.blm, _3v51_Lucene410_0.dvm, _3v51.nvm, _3v51.si, _3v52.fdt, _3v52_es090_0.pos, _3v52_es090_0.tip, _3v52_Lucene410_0.dvd, _3v52.nvd, _3v52.fnm, _3v53.fdt, _3v53_es090_0.doc, _3v53_es090_0.pos, _3v53_es090_0.tim, _3v53_es090_0.tip, _3v53_es090_0.blm, _3v53_Lucene410_0.dvd, _3v53_Lucene410_0.dvm, _3v53.nvd, _3v53.nvm, _3v53.fnm, _3v53.si, _3v4p.fdx, _3v4p.fdt, _3v4o.fnm, _3v4o.si, _3v4n.fdx, _3v4n.fdt, _3v4p_es090_0.doc, _3v4p_es090_0.pos, _3v4p_es090_0.tim, _3v4p_es090_0.tip, _3v4n_es090_0.doc, _3v4n_es090_0.pos, _3v4n_es090_0.tim, _3v4n_es090_0.tip, _3v4p_es090_0.blm, _3v4p_Lucene410_0.dvd, _3v4p_Lucene410_0.dvm, _3v4p.nvd, _3v4p.nvm, _3v4n_es090_0.blm, _3v4n_Lucene410_0.dvd, _3v4n_Lucene410_0.dvm, _3v4n.nvd, _3v4n.nvm, _3v4n.fnm, _3v4n.si, _3v4m.fdx, _3v4m.fdt, _3v4p.fnm, _3v4p.si, _3v4m_es090_0.doc, _3v4m_es090_0.pos, _3v4m_es090_0.tim, _3v4m_es090_0.tip, _3v4m_es090_0.blm, _3v4m_Lucene410_0.dvd, _3v4m_Lucene410_0.dvm, _3v4m.nvd, _3v4m.nvm, segments_3u6, recovery.1419363916358._3v4u_es090_0.blm, recovery.1419363916358._3v4v_es090_0.pos, recovery.1419363916358._3v4v_Lucene410_0.dvd, recovery.1419363916358._3v4v_Lucene410_0.dvm, recovery.1419363916358._3v4v_es090_0.tip, recovery.1419363916358._3v4v_es090_0.blm, recovery.1419363916358._3v4v_es090_0.doc, recovery.1419363916358._3v4v_es090_0.tim, recovery.1419363916358._3v4w_es090_0.doc, recovery.1419363916358._3v4w_Lucene410_0.dvm, recovery.1419363916358._3v4w_es090_0.pos, recovery.1419363916358._3v4w_Lucene410_0.dvd, recovery.1419363916358._3v4w_es090_0.blm, recovery.1419363916358._3v4w_es090_0.tim, recovery.1419363916358._3v4w_es090_0.tip, recovery.1419363916358._3v4x_Lucene410_0.dvd, recovery.1419363916358._3v4x_es090_0.tim, recovery.1419363916358._3v4x_es090_0.pos, recovery.1419363916358._3v4x_es090_0.tip, recovery.1419363916358._3v4x_es090_0.doc, recovery.1419363916358._3v4x_es090_0.blm, recovery.1419363916358._3v4x.si, recovery.1419363916358._3v4x.nvm, recovery.1419363916358._3v4x.fnm, recovery.1419363916358._3v4x.fdt, recovery.1419363916358._3v4x_Lucene410_0.dvm, recovery.1419363916358._3v4y.fdx, recovery.1419363916358._3v4x.nvd, recovery.1419363916358._3v4y_Lucene410_0.dvm, recovery.1419363916358._3v4y.nvd, recovery.1419363916358._3v4y_es090_0.tim, recovery.1419363916358._3v4y_es090_0.tip, recovery.1419363916358._3v4y_es090_0.blm, recovery.1419363916358._3v4y.nvm, recovery.1419363916358._3v4y_es090_0.doc, recovery.1419363916358._3v4y_es090_0.pos, recovery.1419363916358._3v4y.si, recovery.1419363916358._3v4y_Lucene410_0.dvd, recovery.1419363916358._3v4y.fnm, recovery.1419363916358._3v4y.fdt, recovery.1419363916358._3v4h.si, recovery.1419363916358._3v4i.cfs, recovery.1419363916358._3v4i.cfe, recovery.1419363916358._3v4i.si, recovery.1419363916358._3v4p_Lucene410_0.dvm, recovery.1419363916358._3v4p.fnm, recovery.1419363916358._3v4p.nvm, recovery.1419363916358._3v4p_es090_0.blm, recovery.1419363916358._3v4p_Lucene410_0.dvd, recovery.1419363916358._3v4p_es090_0.tim, recovery.1419363916358._3v4p_es090_0.tip, recovery.1419363916358._3v4p.si, recovery.1419363916358._3v4p.fdx, recovery.1419363916358._3v4p_es090_0.doc, recovery.1419363916358._3v4p_es090_0.pos, recovery.1419363916358._3v4n.nvm, recovery.1419363916358._3v4o_Lucene410_0.dvm, recovery.1419363916358._3v4o_es090_0.tip, recovery.1419363916358._3v4o.nvm, recovery.1419363916358._3v4o_es090_0.doc, recovery.1419363916358._3v4o_es090_0.pos, recovery.1419363916358._3v4o_es090_0.tim, recovery.1419363916358._3v4o_Lucene410_0.dvd, recovery.1419363916358._3v4o.fnm, recovery.1419363916358._3v4o.si, recovery.1419363916358._3v4o.fdt, recovery.1419363916358._3v4o.nvd, recovery.1419363916358._3v4o_es090_0.blm, recovery.1419363916358._3v4o.fdx, recovery.1419363916358._3v4l.si, recovery.1419363916358._3v4m_Lucene410_0.dvm, recovery.1419363916358._3v4m_Lucene410_0.dvd, recovery.1419363916358._3v4m.fdx, recovery.1419363916358._3v4m.si, recovery.1419363916358._3v4m.fnm, recovery.1419363916358._3v4m.nvm, recovery.1419363916358._3v4m_es090_0.tip, recovery.1419363916358._3v4m_es090_0.blm, recovery.1419363916358._3v4k.si, recovery.1419363916358._3v4n.fdt, _3v4j_es090_0.blm, _3v4j.fdx, _3v4j_es090_0.pos, _3v4j_Lucene410_0.dvd, _3v4j.nvm, _3v4k.fnm, _3v4k_Lucene410_0.dvd, _3v4k.fdx, _3v4k_es090_0.tip, _3v4k_Lucene410_0.dvm, _3v4k_es090_0.blm, _3v4k.nvm, _3v4e_1.del, _3v02_2.del, _3v1c_4.del, _3uz0_1.del, segments_3sh, recovery.1419363916358._3v4n_es090_0.pos, recovery.1419363916358._3v4m.nvd, recovery.1419363916358._3v4m_es090_0.doc, recovery.1419363916358._3v4m_es090_0.pos, recovery.1419363916358._3v4m.fdt, recovery.1419363916358._3v4m_es090_0.tim, _3v4l.nvd, _3v4l_es090_0.pos, _3v4l.fdt, _3v4l_es090_0.doc, _3v4l_es090_0.tim, _3v4j_es090_0.doc, _3v4j.fdt, _3v4j.nvd, _3v4j_es090_0.tim, _3v4k.nvd, _3v4k.fdt, _3v4k_es090_0.pos, _3v4k_es090_0.doc, _3v4k_es090_0.tim]]; nested: FileNotFoundException[No such file [_3v4q.si]]; ]]
[2014-12-23 19:47:08,048][WARN ][indices.cluster          ] [ESD_dev_i-438764ae] [element_v1][9] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [element_v1][9] failed to fetch index version after copying it over
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:158)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [element_v1][9] shard allocated for local recovery (post api), should exist, but doesn't, current files: [recovery.1419363916358._3v4z_es090_0.tim, _3v1c.nvd, _3v1g_2.del, recovery.1419363916358._3uza_Lucene410_0.dvd, recovery.1419363916358._3v50_Lucene410_0.dvd, recovery.1419363916358._3v50_Lucene410_0.dvm, recovery.1419363916358._3v50_es090_0.tip, recovery.1419363916358._3v4n_es090_0.tim, recovery.1419363916358._3v50.fdx, recovery.1419363916358._3v4z_Lucene410_0.dvm, recovery.1419363916358._3v4z_es090_0.doc, _3v02.fdt, _3v0q_3.del, _3v4m.fnm, _3v4m.si, recovery.1419363916358._3v4r.fdx, segments_3u8, segments.gen, recovery.1419363916358._3uza.si, recovery.1419363916358._3uza_es090_0.tip, recovery.1419363916358._3uza_es090_0.doc, recovery.1419363916358._3v4p.nvd, recovery.1419363916358._3uza.fnm, recovery.1419363916358._3v4n.nvd, recovery.1419363916358._3v4p.fdt, recovery.1419363916358._3uza_es090_0.pos, recovery.1419363916358._3uza_Lucene410_0.dvm, recovery.1419363916358._3uza.fdt, recovery.1419363916358._3uza_es090_0.tim, recovery.1419363916358._3uza_es090_0.blm, recovery.1419363916358._3v50.fnm, recovery.1419363916358._3v50.fdt, recovery.1419363916358._3v50_es090_0.tim, recovery.1419363916358._3v50_es090_0.pos, recovery.1419363916358._3v4z_es090_0.pos, recovery.1419363916358._3v4z.nvm, recovery.1419363916358._3v4z.fnm, _3v02.nvd, _3v1g.fnm, recovery.1419363916358._3v4q_Lucene410_0.dvm, _3v4h.fdt, recovery.1419363916358._3v4z.nvd, recovery.1419363916358._3v4r.nvd, _3v1g.fdt, recovery.1419363916358._3v4z_es090_0.tip, _3v1c_es090_0.blm, recovery.1419363916358._3v4z.fdt, recovery.1419363916358._3v4z_es090_0.blm, _3v1g_es090_0.doc, _3v1c.si, _3v1g_es090_0.pos, _3v1c_Lucene410_0.dvm, _3v1g_es090_0.tim, _3v1c.fdt, _3v1g.fdx, recovery.1419363916358._3v4r.si, _3v1g_Lucene410_0.dvm, _3v1c_es090_0.pos, _3v1g.nvd, _3v1c_Lucene410_0.dvd, _3v1g.nvm, _3v1c_es090_0.tip, _3v1g_es090_0.tip, _3v1c.fnm, _3v0q.nvd, _3v4h.nvd, _3v1c_es090_0.doc, _3v0q_es090_0.tim, recovery.1419363916358._3v4z.fdx, recovery.1419363916358._3v4r.fdt, _3v1g_es090_0.blm, _3v02.fnm, _3v4o.fdx, _3v4l.cfs, _3v02.fdx, _3v1g_Lucene410_0.dvd, _3v02_Lucene410_0.dvm, _3v0q.fdt, _3v1c.fdx, _3v4h.fdx, recovery.1419363916358._3v4r.fnm, _3v0q.fdx, _3v1c.nvm, _3v4l.cfe, _3uzk_1.del, _3v02_es090_0.tim, _3v0q_es090_0.doc, _3v02_es090_0.doc, _3v0q_es090_0.pos, recovery.1419363916358._3uza.nvd, recovery.1419363916358._3uza.fdx, recovery.1419363916358._3uza.nvm, recovery.1419363916358._3v50_es090_0.blm, recovery.1419363916358._3v50.nvm, recovery.1419363916358._3v50.si, recovery.1419363916358._3v50.nvd, recovery.1419363916358._3v50_es090_0.doc, recovery.1419363916358._3v4z.si, recovery.1419363916358._3v4z_Lucene410_0.dvd, _3v1c_es090_0.tim, recovery.1419363916358._3v4t_es090_0.tim, _3v4h.fnm, _3v4o.nvd, recovery.1419363916358._3v4n_Lucene410_0.dvd, recovery.1419363916358._3v4t.nvm, _3v4h_es090_0.pos, recovery.1419363916358._3v4t.fdx, _3v4o.nvm, _3uz0.fdx, _3v4h_Lucene410_0.dvm, _3uz0.nvd, _3v4h_es090_0.tim, _3uz0.fdt, _3v4h_es090_0.doc, _3v02_es090_0.tip, _3v4q_Lucene410_0.dvm, recovery.1419363916358._3v4q_Lucene410_0.dvd, _3uz0_Lucene410_0.dvm, _3v4o.fdt, recovery.1419363916358._3v4n_es090_0.blm, _3v4o_es090_0.doc, _3uz0.fnm, _3v4o_es090_0.pos, _3v02_es090_0.blm, _3v4l.si, recovery.1419363916358._3v4t.si, _3v4o_es090_0.tim, _3v02_es090_0.pos, _3v4h.nvm, _3v02.si, _3v4o_es090_0.tip, recovery.1419363916358._3v4n.fdx, recovery.1419363916358._3v4t.nvd, recovery.1419363916358._3v4n_es090_0.tip, _3v4h_es090_0.tip, _3uz0.si, _3v4q_Lucene410_0.dvd, recovery.1419363916358._3v4n_Lucene410_0.dvm, _3v02_Lucene410_0.dvd, _3v4h_es090_0.blm, _3v02.nvm, _3v4h_Lucene410_0.dvd, recovery.1419363916358._3v4n.fnm, recovery.1419363916358._3v4t.fnm, _17sq_es090_0.tip, _3v4h.cfs, _17sq_es090_0.tim, _3v4h.cfe, _17sq.nvd, _3uz0_es090_0.tip, _17sq_es090_0.blm, _3uz0_es090_0.tim, _17sq.nvm, _3uz0_es090_0.doc, _17sq_es090_0.doc, _3uza.fdx, _17sq.si, _3uzk.fdx, _17sq.fdx, _3v0q_es090_0.tip, _17sq.fdt, _3uza_1.del, _17sq.fnm, _3uza.fdt, _17sq_Lucene45_0.dvd, _3v0q_es090_0.blm, _17sq_Lucene45_0.dvm, _3uz0_Lucene410_0.dvd, _17sq_es090_0.pos, recovery.1419363916358._3v4r_es090_0.tim, recovery.1419363916358._3v4r_es090_0.doc, recovery.1419363916358._3v4r_es090_0.tip, recovery.1419363916358._3v4r_es090_0.pos, recovery.1419363916358._3v4r_es090_0.blm, recovery.1419363916358._3v4r_Lucene410_0.dvd, recovery.1419363916358._3v4r.nvm, recovery.1419363916358._3v4s.fdx, recovery.1419363916358._3v4s_es090_0.tip, recovery.1419363916358._3v4s_es090_0.blm, recovery.1419363916358._3v4s_Lucene410_0.dvm, recovery.1419363916358._3v4s_es090_0.doc, recovery.1419363916358._3v4s_Lucene410_0.dvd, recovery.1419363916358._3v4t.fdt, recovery.1419363916358._3v4t_es090_0.pos, _3v0q_Lucene410_0.dvd, _3uz0_es090_0.blm, _3uzk.fdt, recovery.1419363916358._3v4u.nvd, _3uzk_es090_0.doc, recovery.1419363916358._3v4n_es090_0.doc, recovery.1419363916358._3v4r_Lucene410_0.dvm, recovery.1419363916358._3v4s.fnm, recovery.1419363916358._3v4s.fdt, recovery.1419363916358._3v4s_es090_0.tim, recovery.1419363916358._3v4s.nvd, recovery.1419363916358._3v4s.si, recovery.1419363916358._3v4s.nvm, recovery.1419363916358._3v4s_es090_0.pos, recovery.1419363916358._3v4t_Lucene410_0.dvd, recovery.1419363916358._3v4v.si, _3v4i.cfs, _3v4j.fnm, _3v4i.cfe, recovery.1419363916358._3v4u_es090_0.doc, recovery.1419363916358._3v4v.nvm, _3uzk_Lucene410_0.dvm, _3uz0_es090_0.pos, _checksums-1419291672870, _3v4l_es090_0.tip, _3v4o_Lucene410_0.dvd, _3v4l.fnm, _3uzk.nvd, _3uz0.nvm, _3uzk.nvm, _3v4f.si, _3uzk.fnm, recovery.1419363916358._3v4t_es090_0.tip, recovery.1419363916358._3v4t_es090_0.blm, recovery.1419363916358._3v4t_es090_0.doc, recovery.1419363916358._3v4u_Lucene410_0.dvd, _3v4l_Lucene410_0.dvm, _17sq_27.del, recovery.1419363916358._3v4u_es090_0.tim, recovery.1419363916358._3v4u.fdt, _3v0q.nvm, recovery.1419363916358._3v4u_es090_0.tip, recovery.1419363916358._3v4u.si, _3v4o_Lucene410_0.dvm, _3v4f.cfs, _3v0q_Lucene410_0.dvm, recovery.1419363916358._3v4t_Lucene410_0.dvm, recovery.1419363916358._3v4u_Lucene410_0.dvm, recovery.1419363916358._3v4u.fdx, write.lock, recovery.1419363916358._3v4v.fdx, recovery.1419363916358._3v4n.si, _3v4r.fdx, recovery.1419363916358._3v4v.nvd, _3v4r.fdt, recovery.1419363916358._3v4v.fnm, _3v4r_es090_0.doc, _3v4l_es090_0.blm, _3v4r_es090_0.pos, _3v4l.nvm, _3v4r_es090_0.tim, _3v4e.nvm, _3uzk.si, recovery.1419363916358._3v4v.fdt, _3v4r_es090_0.tip, _3v4f.cfe, _3uza_es090_0.pos, _3v4e_es090_0.tip, _3uza.fnm, _3v4e.nvd, _3uza.si, _3v4e_es090_0.tim, _3uza_es090_0.tim, _3v4e_Lucene410_0.dvd, _3v4r_es090_0.blm, recovery.1419363916358._3v4u.nvm, recovery.1419363916358._3v4u_es090_0.pos, _3uza_es090_0.tip, _3v4l_Lucene410_0.dvd, _3v4r_Lucene410_0.dvd, _3v4j_es090_0.tip, _3v4o_es090_0.blm, _3v4e_es090_0.doc, _3v4h_1.del, recovery.1419363916358._3v4w.nvd, _3v4r_Lucene410_0.dvm, _3v4j.si, _3v4h.si, _3v4e.fnm, _3v4r.nvd, _3v4e_Lucene410_0.dvm, _3v1g.si, recovery.1419363916358._3v4w.fdx, _3v4r.nvm, recovery.1419363916358._3v4w.nvm, _3uzk_es090_0.pos, _3uzk_es090_0.tim, _3uzk_es090_0.tip, _3uzk_es090_0.blm, _3v4i.si, _3uza_es090_0.doc, _3uzk_Lucene410_0.dvd, recovery.1419363916358._3v4w.fnm, _3v4r.fnm, recovery.1419363916358._3v4w.fdt, _3v4r.si, _3v4e.fdt, _3v4s.fdx, recovery.1419363916358._3v4w.si, _3v4s.fdt, recovery.1419363916358._3v4x.fdx, _3v4s_es090_0.doc, _3v4j_Lucene410_0.dvm, _3v4s_es090_0.pos, _3v4e_es090_0.blm, _3v0q.fnm, _3v4e.fdx, _3v4s_es090_0.tim, _3v4e.si, _3v4s_es090_0.tip, recovery.1419363916358._3v4u.fnm, _3v4e_es090_0.pos, _3v4s.nvd, _3v0q.si, _3v4l.fdx, _3v4s.nvm, _3v52.si, _3v4k.cfs, _3v4k.cfe, _3v4k.si, _3v4s_Lucene410_0.dvd, _3v4s.fnm, _3v4s.si, _3v4t.fdt, _3v4t_es090_0.pos, _3v4t_es090_0.tip, _3v4t_Lucene410_0.dvd, _3v4t.nvd, _3v4t.fnm, _3v4u.fdx, _3v4u_es090_0.doc, _3v4u_es090_0.tim, _3v4u_es090_0.blm, _3v4u_Lucene410_0.dvm, _3v4u.nvm, _3v4u.si, _3v4v.fdt, _3v4v_es090_0.pos, _3v4v_es090_0.tip, _3v4v_Lucene410_0.dvd, _3v4v.nvd, _3v4v.fnm, _3v4w.fdx, _3v4w_es090_0.doc, _3v4w_es090_0.tim, _3v4w_es090_0.blm, _3v4w_Lucene410_0.dvm, _3v4w.nvm, _3v4w.si, _3v4x.fdt, _3v4x_es090_0.pos, _3v4x_es090_0.tip, _3v4x_Lucene410_0.dvd, _3v4x.nvd, _3v4x.fnm, _3v4y.fdx, _3v4y_es090_0.doc, _3v4y_es090_0.tim, _3v4y_es090_0.blm, _3v4y_Lucene410_0.dvm, _3v4y.nvm, _3v4y.si, _3v4z.fdt, _3v4z_es090_0.pos, _3v4z_es090_0.tip, _3v4z_Lucene410_0.dvd, _3v4z.nvd, _3v4z.fnm, _3v50.fdx, _3v50_es090_0.doc, _3v50_es090_0.tim, _3v50_es090_0.blm, _3v50_Lucene410_0.dvm, _3v50.nvm, _3v50.si, _3v51.fdt, _3v51_es090_0.pos, _3v51_es090_0.tip, _3v51_Lucene410_0.dvd, _3v51.nvd, _3v51.fnm, _3v52.fdx, _3v52_es090_0.doc, _3v52_es090_0.tim, _3v52_es090_0.blm, _3v52_Lucene410_0.dvm, _3v52.nvm, _3v53.fdx, _3uza_es090_0.blm, _3uza_Lucene410_0.dvd, _3uza_Lucene410_0.dvm, _3uza.nvd, _3uza.nvm, _3v4s_es090_0.blm, _3v4s_Lucene410_0.dvm, _3v4t.fdx, _3v4t_es090_0.doc, _3v4t_es090_0.tim, _3v4t_es090_0.blm, _3v4t_Lucene410_0.dvm, _3v4t.nvm, _3v4t.si, _3v4u.fdt, _3v4u_es090_0.pos, _3v4u_es090_0.tip, _3v4u_Lucene410_0.dvd, _3v4u.nvd, _3v4u.fnm, _3v4v.fdx, _3v4v_es090_0.doc, _3v4v_es090_0.tim, _3v4v_es090_0.blm, _3v4v_Lucene410_0.dvm, _3v4v.nvm, _3v4v.si, _3v4w.fdt, _3v4w_es090_0.pos, _3v4w_es090_0.tip, _3v4w_Lucene410_0.dvd, _3v4w.nvd, _3v4w.fnm, _3v4x.fdx, _3v4x_es090_0.doc, _3v4x_es090_0.tim, _3v4x_es090_0.blm, _3v4x_Lucene410_0.dvm, _3v4x.nvm, _3v4x.si, _3v4y.fdt, _3v4y_es090_0.pos, _3v4y_es090_0.tip, _3v4y_Lucene410_0.dvd, _3v4y.nvd, _3v4y.fnm, _3v4z.fdx, _3v4z_es090_0.doc, _3v4z_es090_0.tim, _3v4z_es090_0.blm, _3v4z_Lucene410_0.dvm, _3v4z.nvm, _3v4z.si, _3v50.fdt, _3v50_es090_0.pos, _3v50_es090_0.tip, _3v50_Lucene410_0.dvd, _3v50.nvd, _3v50.fnm, _3v51.fdx, _3v51_es090_0.doc, _3v51_es090_0.tim, _3v51_es090_0.blm, _3v51_Lucene410_0.dvm, _3v51.nvm, _3v51.si, _3v52.fdt, _3v52_es090_0.pos, _3v52_es090_0.tip, _3v52_Lucene410_0.dvd, _3v52.nvd, _3v52.fnm, _3v53.fdt, _3v53_es090_0.doc, _3v53_es090_0.pos, _3v53_es090_0.tim, _3v53_es090_0.tip, _3v53_es090_0.blm, _3v53_Lucene410_0.dvd, _3v53_Lucene410_0.dvm, _3v53.nvd, _3v53.nvm, _3v53.fnm, _3v53.si, _3v4p.fdx, _3v4p.fdt, _3v4o.fnm, _3v4o.si, _3v4n.fdx, _3v4n.fdt, _3v4p_es090_0.doc, _3v4p_es090_0.pos, _3v4p_es090_0.tim, _3v4p_es090_0.tip, _3v4n_es090_0.doc, _3v4n_es090_0.pos, _3v4n_es090_0.tim, _3v4n_es090_0.tip, _3v4p_es090_0.blm, _3v4p_Lucene410_0.dvd, _3v4p_Lucene410_0.dvm, _3v4p.nvd, _3v4p.nvm, _3v4n_es090_0.blm, _3v4n_Lucene410_0.dvd, _3v4n_Lucene410_0.dvm, _3v4n.nvd, _3v4n.nvm, _3v4n.fnm, _3v4n.si, _3v4m.fdx, _3v4m.fdt, _3v4p.fnm, _3v4p.si, _3v4m_es090_0.doc, _3v4m_es090_0.pos, _3v4m_es090_0.tim, _3v4m_es090_0.tip, _3v4m_es090_0.blm, _3v4m_Lucene410_0.dvd, _3v4m_Lucene410_0.dvm, _3v4m.nvd, _3v4m.nvm, segments_3u6, recovery.1419363916358._3v4u_es090_0.blm, recovery.1419363916358._3v4v_es090_0.pos, recovery.1419363916358._3v4v_Lucene410_0.dvd, recovery.1419363916358._3v4v_Lucene410_0.dvm, recovery.1419363916358._3v4v_es090_0.tip, recovery.1419363916358._3v4v_es090_0.blm, recovery.1419363916358._3v4v_es090_0.doc, recovery.1419363916358._3v4v_es090_0.tim, recovery.1419363916358._3v4w_es090_0.doc, recovery.1419363916358._3v4w_Lucene410_0.dvm, recovery.1419363916358._3v4w_es090_0.pos, recovery.1419363916358._3v4w_Lucene410_0.dvd, recovery.1419363916358._3v4w_es090_0.blm, recovery.1419363916358._3v4w_es090_0.tim, recovery.1419363916358._3v4w_es090_0.tip, recovery.1419363916358._3v4x_Lucene410_0.dvd, recovery.1419363916358._3v4x_es090_0.tim, recovery.1419363916358._3v4x_es090_0.pos, recovery.1419363916358._3v4x_es090_0.tip, recovery.1419363916358._3v4x_es090_0.doc, recovery.1419363916358._3v4x_es090_0.blm, recovery.1419363916358._3v4x.si, recovery.1419363916358._3v4x.nvm, recovery.1419363916358._3v4x.fnm, recovery.1419363916358._3v4x.fdt, recovery.1419363916358._3v4x_Lucene410_0.dvm, recovery.1419363916358._3v4y.fdx, recovery.1419363916358._3v4x.nvd, recovery.1419363916358._3v4y_Lucene410_0.dvm, recovery.1419363916358._3v4y.nvd, recovery.1419363916358._3v4y_es090_0.tim, recovery.1419363916358._3v4y_es090_0.tip, recovery.1419363916358._3v4y_es090_0.blm, recovery.1419363916358._3v4y.nvm, recovery.1419363916358._3v4y_es090_0.doc, recovery.1419363916358._3v4y_es090_0.pos, recovery.1419363916358._3v4y.si, recovery.1419363916358._3v4y_Lucene410_0.dvd, recovery.1419363916358._3v4y.fnm, recovery.1419363916358._3v4y.fdt, recovery.1419363916358._3v4h.si, recovery.1419363916358._3v4i.cfs, recovery.1419363916358._3v4i.cfe, recovery.1419363916358._3v4i.si, recovery.1419363916358._3v4p_Lucene410_0.dvm, recovery.1419363916358._3v4p.fnm, recovery.1419363916358._3v4p.nvm, recovery.1419363916358._3v4p_es090_0.blm, recovery.1419363916358._3v4p_Lucene410_0.dvd, recovery.1419363916358._3v4p_es090_0.tim, recovery.1419363916358._3v4p_es090_0.tip, recovery.1419363916358._3v4p.si, recovery.1419363916358._3v4p.fdx, recovery.1419363916358._3v4p_es090_0.doc, recovery.1419363916358._3v4p_es090_0.pos, recovery.1419363916358._3v4n.nvm, recovery.1419363916358._3v4o_Lucene410_0.dvm, recovery.1419363916358._3v4o_es090_0.tip, recovery.1419363916358._3v4o.nvm, recovery.1419363916358._3v4o_es090_0.doc, recovery.1419363916358._3v4o_es090_0.pos, recovery.1419363916358._3v4o_es090_0.tim, recovery.1419363916358._3v4o_Lucene410_0.dvd, recovery.1419363916358._3v4o.fnm, recovery.1419363916358._3v4o.si, recovery.1419363916358._3v4o.fdt, recovery.1419363916358._3v4o.nvd, recovery.1419363916358._3v4o_es090_0.blm, recovery.1419363916358._3v4o.fdx, recovery.1419363916358._3v4l.si, recovery.1419363916358._3v4m_Lucene410_0.dvm, recovery.1419363916358._3v4m_Lucene410_0.dvd, recovery.1419363916358._3v4m.fdx, recovery.1419363916358._3v4m.si, recovery.1419363916358._3v4m.fnm, recovery.1419363916358._3v4m.nvm, recovery.1419363916358._3v4m_es090_0.tip, recovery.1419363916358._3v4m_es090_0.blm, recovery.1419363916358._3v4k.si, recovery.1419363916358._3v4n.fdt, _3v4j_es090_0.blm, _3v4j.fdx, _3v4j_es090_0.pos, _3v4j_Lucene410_0.dvd, _3v4j.nvm, _3v4k.fnm, _3v4k_Lucene410_0.dvd, _3v4k.fdx, _3v4k_es090_0.tip, _3v4k_Lucene410_0.dvm, _3v4k_es090_0.blm, _3v4k.nvm, _3v4e_1.del, _3v02_2.del, _3v1c_4.del, _3uz0_1.del, segments_3sh, recovery.1419363916358._3v4n_es090_0.pos, recovery.1419363916358._3v4m.nvd, recovery.1419363916358._3v4m_es090_0.doc, recovery.1419363916358._3v4m_es090_0.pos, recovery.1419363916358._3v4m.fdt, recovery.1419363916358._3v4m_es090_0.tim, _3v4l.nvd, _3v4l_es090_0.pos, _3v4l.fdt, _3v4l_es090_0.doc, _3v4l_es090_0.tim, _3v4j_es090_0.doc, _3v4j.fdt, _3v4j.nvd, _3v4j_es090_0.tim, _3v4k.nvd, _3v4k.fdt, _3v4k_es090_0.pos, _3v4k_es090_0.doc, _3v4k_es090_0.tim]
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:132)
    ... 4 more
Caused by: java.io.FileNotFoundException: No such file [_3v4q.si]
    at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
    at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
    at org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
    at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:487)
    at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
    at org.apache.lucene.codecs.lucene46.Lucene46SegmentInfoReader.read(Lucene46SegmentInfoReader.java:51)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:358)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:454)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:906)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:85)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:123)
    ... 4 more
clintongormley commented 8 years ago

So much has changed since 1.4.2, closing