Test failure: QA integration test - Elasticsearch red or offline

[jordansissel]

https://logstash-ci.elastic.co/job/elastic+logstash+5.6+multijob-integration-1/94/console

21:34:36   1) Test Logstash instance should keep the same id between restarts
21:34:36      Failure/Error: JSON.parse(open("http://localhost:9600/").read)["id"]
21:34:36      
21:34:36      Errno::ECONNREFUSED:
21:34:36        Connection refused - Connection refused
21:34:36      # ./specs/01_logstash_bin_smoke_spec.rb:165:in `get_id'
21:34:36      # ./specs/01_logstash_bin_smoke_spec.rb:177:in `(root)'
21:34:36      # ./specs/01_logstash_bin_smoke_spec.rb:177:in `(root)'
21:34:36 
21:34:36   2) Test Dead Letter Queue using logstash.yml and separate config file behaves like it can send 1000 documents to and index from the dlq should index all documents
21:34:36      Failure/Error: result = es_client.search(index: 'logstash-*', size: 0, q: '*')
21:34:36      
21:34:36      Elasticsearch::Transport::Transport::Errors::ServiceUnavailable:
21:34:36        [503] {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[],"caused_by":{"type":"shard_not_found_exception","reason":"no such shard","index_uuid":"fj_PKqiORbqlm6vrH-Msdw","shard":"2","index":"logstash-2018.01.02"}},"status":503}
21:34:36      Shared Example Group: "it can send 1000 documents to and index from the dlq" called from ./specs/dlq_spec.rb:64
21:34:36      # ./specs/dlq_spec.rb:48:in `(root)'
21:34:36      # ./specs/dlq_spec.rb:47:in `(root)'
21:34:36 
21:34:36 Finished in 20 minutes 6 seconds (files took 11.79 seconds to load)
21:34:36 21 examples, 2 failures

[jordansissel]

Same as https://logstash-ci.elastic.co/job/elastic+logstash+5.6+multijob-integration-pq-1/94/console

21:34:31   1) Test Logstash instance should keep the same id between restarts
21:34:31      Failure/Error: JSON.parse(open("http://localhost:9600/").read)["id"]
21:34:31      
21:34:31      Errno::ECONNREFUSED:
21:34:31        Connection refused - Connection refused
21:34:31      # ./specs/01_logstash_bin_smoke_spec.rb:165:in `get_id'
21:34:31      # ./specs/01_logstash_bin_smoke_spec.rb:177:in `(root)'
21:34:31      # ./specs/01_logstash_bin_smoke_spec.rb:177:in `(root)'
21:34:31 
21:34:31   2) Test Dead Letter Queue using logstash.yml and separate config file behaves like it can send 1000 documents to and index from the dlq should index all documents
21:34:31      Failure/Error: result = es_client.search(index: 'logstash-*', size: 0, q: '*')
21:34:31      
21:34:31      Elasticsearch::Transport::Transport::Errors::ServiceUnavailable:
21:34:31        [503] {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[],"caused_by":{"type":"illegal_index_shard_state_exception","reason":"CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]","index_uuid":"EirBXJApQBawgq9hqRBATA","shard":"1","index":"logstash-2018.01.02"}},"status":503}
21:34:31      Shared Example Group: "it can send 1000 documents to and index from the dlq" called from ./specs/dlq_spec.rb:64
21:34:31      # ./specs/dlq_spec.rb:48:in `(root)'
21:34:31      # ./specs/dlq_spec.rb:47:in `(root)'
21:34:31 
21:34:31 Finished in 20 minutes 1 second (files took 11.43 seconds to load)
21:34:31 21 examples, 2 failures