"Unhandled exception - Errno::ECONNREFUSED: Connection refused - Connection refused"

[techraf]

On a fresh Debian 8 VM, after the setup (no problems at all) I get continuous errors in /var/log/logstash/logstash.log.

{:timestamp=>"2016-08-06T23:21:19.397000+0900", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-08-06T23:23:44.497000+0900", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2016-08-06T23:23:44.509000+0900", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2016-08-06T23:23:44.518000+0900", :message=>"Unhandled exception", :error=>#<Errno::ECONNREFUSED: Connection refused - Connection refused>, :level=>:error}
{:timestamp=>"2016-08-06T23:23:44.556000+0900", :message=>"Unhandled exception", :error=>#<Errno::ECONNREFUSED: Connection refused - Connection refused>, :level=>:error}
{:timestamp=>"2016-08-06T23:23:44.622000+0900", :message=>"Unhandled exception", :error=>#<Errno::ECONNREFUSED: Connection refused - Connection refused>, :level=>:error}

At the same time on Molecule the same log file after setting up ansible-role-elk looks like this:

{:timestamp=>"2016-08-06T05:13:50.329000+0000", :message=>"Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads", :count_was=>2, :filters=>["multiline", "multiline", "throttle"], :level=>:warn}
{:timestamp=>"2016-08-06T05:26:16.659000+0000", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-08-06T05:26:42.340000+0000", :message=>"Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads", :count_was=>2, :filters=>["multiline", "multiline", "throttle"], :level=>:warn}

Yet on another clone of the separate VM if I set up geerlingguy.logstash (with its dependencies), I get only the following in /var/log/logstash/logstash.log:

{:timestamp=>"2016-08-06T23:29:18.499000+0900", :message=>"Pipeline main started"}

Neither installation works for me so far. I am lost about the meaning and consequences.

[conorsch]

Thanks for the well-written report, will try to reproduce locally and follow up.

[conorsch]

@techraf Do you have any clients shipping logs to the test VM? From the error output you posted, it looks like the elasticsearch service isn't running, but the tests should catch that if it's down.

Take a look at https://github.com/freedomofpress/logserver-integration for a sandboxed setup that will ship logs from a separate VM. Simply run make test after cloning that repo, then you should be able to log in at http://127.0.0.1:8080 to the Kibana web interface and confirm that logs are shipping correctly.