search

fabric8io / openshift-elasticsearch-plugin

Apache License 2.0
27 stars 21 forks source link

Updating the to waiting for cluster to be up instead of in state yell… #40

Closed ewolinetz closed 8 years ago

ewolinetz commented 8 years ago

…ow so we can still seed SG even if the cluster cant recover fully

Addresses #39

ewolinetz commented 8 years ago

@jcantrill PTAL

ewolinetz commented 8 years ago 
[2016-09-08 15:26:04,162][WARN ][indices.cluster          ] [Conan the Barbarian] [[rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][0]] marking and sending shard failed due to [failed recovery]
[rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][[rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][0]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si];
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)
    at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
    at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: [rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][[rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][0]] IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si];
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:208)
    ... 5 more
Caused by: java.nio.file.NoSuchFileException: /elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
    at java.nio.channels.FileChannel.open(FileChannel.java:287)
    at java.nio.channels.FileChannel.open(FileChannel.java:335)
    at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
    at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:186)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89)
    at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109)
    at org.apache.lucene.codecs.lucene50.Lucene50SegmentInfoFormat.read(Lucene50SegmentInfoFormat.java:82)
    at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:362)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:493)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:490)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
    at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:95)
    at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:163)
    at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:148)
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:199)
    ... 5 more
[2016-09-08 15:26:04,174][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[7], s[INITIALIZING], a[id=Tl7kbnI5Qlaa7CyzyuLwgw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.466Z]]], indexUUID [YEH4LxehSgC5Zf0u-Lsf6A], message [failed recovery], failure [IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si]; ]
[rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][[rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][0]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si];
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)
    at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
    at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: [rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][[rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08][0]] IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si];
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:208)
    ... 5 more
Caused by: java.nio.file.NoSuchFileException: /elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.d7dec85c-75d1-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
    at java.nio.channels.FileChannel.open(FileChannel.java:287)
    at java.nio.channels.FileChannel.open(FileChannel.java:335)
    at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
    at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:186)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89)
    at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109)
    at org.apache.lucene.codecs.lucene50.Lucene50SegmentInfoFormat.read(Lucene50SegmentInfoFormat.java:82)
    at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:362)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:493)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:490)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
    at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:95)
    at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:163)
    at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:148)
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:199)
    ... 5 more
[2016-09-08 15:26:04,382][WARN ][indices.cluster          ] [Conan the Barbarian] [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0]] marking and sending shard failed due to [failed recovery]
[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si];
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)
    at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
    at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0]] IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si];
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:208)
    ... 5 more
Caused by: java.nio.file.NoSuchFileException: /elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
    at java.nio.channels.FileChannel.open(FileChannel.java:287)
    at java.nio.channels.FileChannel.open(FileChannel.java:335)
    at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
    at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:186)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89)
    at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109)
    at org.apache.lucene.codecs.lucene50.Lucene50SegmentInfoFormat.read(Lucene50SegmentInfoFormat.java:82)
    at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:362)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:493)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:490)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
    at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:95)
    at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:163)
    at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:148)
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:199)
    ... 5 more
[2016-09-08 15:26:04,404][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [failed recovery], failure [IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si]; ]
[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si];
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)
    at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
    at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0]] IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock, segments_3]]; nested: NoSuchFileException[/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si];
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:208)
    ... 5 more
Caused by: java.nio.file.NoSuchFileException: /elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08/0/index/_0.si
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
    at java.nio.channels.FileChannel.open(FileChannel.java:287)
    at java.nio.channels.FileChannel.open(FileChannel.java:335)
    at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
    at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:186)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:89)
    at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109)
    at org.apache.lucene.codecs.lucene50.Lucene50SegmentInfoFormat.read(Lucene50SegmentInfoFormat.java:82)
    at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:362)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:493)
    at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:490)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
    at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:95)
    at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:163)
    at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:148)
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:199)
    ... 5 more
[2016-09-08 15:26:04,486][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:05,101][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:05,450][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:05,629][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:05,773][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:05,884][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:05,953][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,000][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,072][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,109][INFO ][cluster.metadata         ] [Conan the Barbarian] [.searchguard.logging-es-l31pwce4-3-zgg1d] creating index, cause [api], templates [], shards [1]/[0], mappings []
[2016-09-08 15:26:06,136][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,195][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,271][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,315][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,372][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,416][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,486][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,557][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,595][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,659][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,704][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,784][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,836][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,912][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
[2016-09-08 15:26:06,937][WARN ][cluster.action.shard     ] [Conan the Barbarian] [rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0] received shard failed for target shard [[rails.7402e6b9-75ce-11e6-9a7c-0efd4172c6af.2016.09.08][0], node[gJtgMtZJQKqfdW3lQu_iqg], [P], v[11], s[INITIALIZING], a[id=AIMcIfT_T1-KS2ggbtWxPQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-09-08T15:26:03.467Z]]], indexUUID [5YY2tYlmQQuEgY9jkmtvRA], message [master {Conan the Barbarian}{gJtgMtZJQKqfdW3lQu_iqg}{172.17.0.3}{logging-es-cluster/172.17.0.3:9300}{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]

[2016-09-08 15:26:07,018][INFO ][cluster.metadata         ] [Conan the Barbarian] [.searchguard.logging-es-l31pwce4-3-zgg1d] create_mapping [config]
[2016-09-08 15:26:07,150][INFO ][cluster.metadata         ] [Conan the Barbarian] [.searchguard.logging-es-l31pwce4-3-zgg1d] create_mapping [roles]
[2016-09-08 15:26:07,192][INFO ][cluster.metadata         ] [Conan the Barbarian] [.searchguard.logging-es-l31pwce4-3-zgg1d] create_mapping [rolesmapping]
[2016-09-08 15:26:07,248][INFO ][cluster.metadata         ] [Conan the Barbarian] [.searchguard.logging-es-l31pwce4-3-zgg1d] create_mapping [actiongroups]
jcantrill commented 8 years ago

LGTM though i dont know the implications of https://github.com/fabric8io/openshift-elasticsearch-plugin/pull/40#issuecomment-245640036 Is it just an example of the output log successfully seeding the ACLs?

ewolinetz commented 8 years ago

@jcantrill correct, an example of output in error is https://github.com/fabric8io/openshift-elasticsearch-plugin/issues/39#issuecomment-245639645

ewolinetz commented 8 years ago

[merge]