A plugin had an unrecoverable error.

[Sunde93]

Hello,

I have this issue when launching logstash, i didn't understand what is the problem :

"A plugin had an unrecoverable error. Will restart this plugin. Plugin: <LogStash::Inputs::Beats port=>5044, type=>"omslog", codec=><LogStash::Codecs::Multiline pattern=>"^%{TIMESTAMP_ISO8601}", negate=>true, what=>"previous", charset=>"UTF-8", multiline_tag=>"multiline", max_lines=>500, max_bytes=>10485760>, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60> Error: event executor terminated {:level=>:error} "

I run logstash with this command : ./bin/logstash -f ./conf

And my configuration's directory contain : [admingl@aude conf]$ ll total 12 -rw-rw-r--. 1 admingl admingl 1760 2 déc. 16:59 02-omslog.conf -rw-rw-r--. 1 admingl admingl 2555 2 déc. 16:59 03-rpulog.conf drwxrwxr-x. 2 admingl admingl 4096 1 déc. 10:53 patterns

[admingl@aude conf]$ cat * input { beats { port => 5044 type => "omslog" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" } } }

filter { if [type] == "omslog" { grok { patterns_dir => ["./patterns"] match => { "message" => "%{TIMESTAMP_ISO8601:[@metadata][timestamp]}( )+%{LOGLEVEL:log-level}( )+%{NUMBER:numproc}( )+[%{GREEDYDATA:container}]( )+---(( )+|( )+.)%{JAVACLASS:class}( )+:( )+%{GREEDYDATA:message}" } overwrite => [ "message" ] } grok { match => { "source" => "\/data\/logs\/%{GREEDYDATA:type_oms}\/%{GREEDYDATA:worker}\/%{GREEDYDATA:logname}" } } date { match => [ "[@metadata][timestamp]", "YYYY-MM-dd HH:mm:ss.SSS" ] } mutate { remove_field => [ "beat.hostname", "beat.name", "offset", "tags" ] }

            if [host] == "talus" or [host] == "pan" {
                    mutate {
                            add_field => {
                                    "env" => "integration"
                            }
                    }
            }

            if [host] == "mustafar" {
                    mutate {
                            add_field => {
                                    "env" => "recette"
                            }
                    }
            }

            if [host] == "nix" {
                    mutate {
                            add_field => {
                                    "env" => "integcontinu"
                            }
                    }
            }

            if [message] !~ /(.+)/  {
                drop { }
            }
    }

}

output { elasticsearch { hosts => "172.16.71.23:9200" index => "omslog-%{+YYYY.MM.dd}" manage_template => false } } input { beats { port => 5044 type => "rpulog" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" } } }

filter { if [type] == "rpulog" { grok { patterns_dir => ["/appli/logstash/conf/patterns"]

2016-11-22 11:44:24 INFO [ajp-bio-8209-exec-1] [FrameworkServlet.java:506] : FrameworkServlet 'dispatcherServlet': initialization completed in 20 ms

                    match => { "message" => "%{TIMESTAMP_ISO8601:[@metadata][timestamp]}( )+%{LOGLEVEL:log-level}( )+\[%{GREEDYDATA:container}\]( )+\[%{JAVACLASS_RPU:class}\]( )+:( )+%{GREEDYDATA:message}" }
                    #2016-12-01 06:02:43,329 [WARN] [checkStocksForPickingJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] Unable to get stocks for Offer with code=56692668 [eligibleStockMut=true]
                    match => { "message" => "%{TIMESTAMP_ISO8601:[@metadata][timestamp]}( )+\[%{LOGLEVEL:log-level}\]( )+\[%{GREEDYDATA:class}::%{JAVACLASS:package-class}\]( )+%{GREEDYDATA:message}" }
                    overwrite => [ "message" ]
            }
            grok {
                    match => { "source" => "\/data\/logs\/%{GREEDYDATA:type_rpu}\/%{GREEDYDATA:logname}" }
                    match => { "source" => "\/appli\/hybris\/%{GREEDYDATA:type_rpu}\/log\/%{GREEDYDATA:logname}" }
            }
            date {
                    match => [ "[@metadata][timestamp]", "YYYY-MM-dd HH:mm:ss" ]
            }
            date {
                    match => [ "[@metadata][timestamp]", "YYYY-MM-dd HH:mm:ss,SSS" ]
            }
            mutate {
                    remove_field => [ "beat.hostname", "beat.name", "offset", "tags" ]
            }

            if [host] == "tatooine" {
                    mutate {
                            add_field => {
                                    "env" => "integration"
                            }
                    }
            }

            if [host] == "taris" {
                    mutate {
                            add_field => {
                                    "env" => "recette"
                            }
                    }
            }

            if [host] == "vortex" {
                    mutate {
                            add_field => {
                                    "env" => "integcontinu"
                            }
                    }
            }

            if [type_rpu] == "hybris" {
                    mutate {
                            update => {
                                    "type_rpu" => "rpu-hybris"
                            }
                    }
            }

            if [message] !~ /(.+)/  {
                drop { }
            }
    }

}

output { elasticsearch { hosts => "172.16.71.23:9200" index => "rpulog-%{+YYYY.MM.dd}" manage_template => false } }

[ph]

This seems to be a similar issue as #170,

I wondering if there is not something going with the multiline implementation in the beats input.

@Sunde93 Can you add the logstash version and the beats version you are currently runnings?

You can get the plugin version with bin/logtash-plugin list --verbose beats

[Sunde93]

Hi, here is my logstash and beat plugin version :

logstash -> 2.4.1 logstash-input-beats -> 3.1.8

Regards,

[runningman84]

this also happens with the latest 5.0.x release of logstash:

[2016-12-09T13:13:58,239][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2016-12-09T13:13:58,318][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2016-12-09T13:14:00,514][INFO ][logstash.outputs.s3      ] Registering s3 output {:bucket=>"dev-log-loggingbucket-xxxxxxxx", :endpoint_region=
>"eu-central-1"}
[2016-12-09T13:14:05,287][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Beats port=>5044, ssl=>true, ssl_certificate=>"/etc/logstash/ssl/cert.pem", ssl_key=>"/etc/logstash/ssl/key.pkcs8", add
_field=>{"logstash_host"=>"i-yyyyyyyyyyyyyyy"}, tags=>["input_beats"], id=>"b5ff663aa135cf7f5e1aa72c21602c3bf5a366e0-1", enable_metric=>true, code
c=><LogStash::Codecs::Plain id=>"plain_b7fc5280-7729-4835-a987-b2edb9edd7c7", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl_verify_mo
de=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60>
  Error: Address already in use
[2016-12-09T13:14:06,292][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2016-12-09T13:14:06,393][WARN ][io.netty.channel.AbstractChannel] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x39697a1b]
java.util.concurrent.RejectedExecutionException: event executor terminated

[jordansissel]

The error you posted says this:

Error: Address already in use

This means that something is already using the port you are telling Logstash to use.

On Fri, Dec 9, 2016 at 6:58 AM runningman84 notifications@github.com wrote:

this also happens with the latest 5.0.x release of logstash:

[2016-12-09T13:13:58,239][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}

[2016-12-09T13:13:58,318][INFO ][org.logstash.beats.Server] Starting server on port: 5044

[2016-12-09T13:14:00,514][INFO ][logstash.outputs.s3 ] Registering s3 output {:bucket=>"dev-log-loggingbucket-xxxxxxxx", :endpoint_region=

"eu-central-1"}

[2016-12-09T13:14:05,287][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.

Plugin: <LogStash::Inputs::Beats port=>5044, ssl=>true, ssl_certificate=>"/etc/logstash/ssl/cert.pem", ssl_key=>"/etc/logstash/ssl/key.pkcs8", add

_field=>{"logstash_host"=>"i-yyyyyyyyyyyyyyy"}, tags=>["input_beats"], id=>"b5ff663aa135cf7f5e1aa72c21602c3bf5a366e0-1", enable_metric=>true, code

c=><LogStash::Codecs::Plain id=>"plain_b7fc5280-7729-4835-a987-b2edb9edd7c7", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl_verify_mo

de=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60>

Error: Address already in use

[2016-12-09T13:14:06,292][INFO ][org.logstash.beats.Server] Starting server on port: 5044

[2016-12-09T13:14:06,393][WARN ][io.netty.channel.AbstractChannel] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x39697a1b]

java.util.concurrent.RejectedExecutionException: event executor terminated

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/logstash-plugins/logstash-input-beats/issues/172#issuecomment-266033244, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIC6gFoq_EZ7DFD_lLfTKbPx_UUS2Ocks5rGWx5gaJpZM4LCth0 .

[runningman84]

@jordansissel yes that was another error... thanks for the hint.

[Sunde93]

Hi, what about my issue ? how can i make it work maybe using another method ?

[Ankit-Kulkarni]

hi @ph I also received the same issue which led me here . Though my issue is solved , I am not sure my finding is of help to you but I could reproduce the below error in logstash version 5.1.1 .

  Plugin: <LogStash::Inputs::Beats port=>5044, id=>"52fbe3a18c2b1a464f627c3b45ee965099dad3c5-4", enable_met
ric=>true, codec=><LogStash::Codecs::Plain id=>"plain_38a36774-734a-477f-bdea-a5d0affd7036", enable_metric=
>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ss
l_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1,
 tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_2
56_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_EC
DHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_
CBC_SHA256"], client_inactivity_timeout=>60>
  Error: event executor terminated
[2017-02-08T05:08:09,427][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2017-02-08T05:08:09,428][WARN ][io.netty.channel.AbstractChannel] Force-closing a channel whose registrati
on task was not accepted by an event loop: [id: 0xde673ed5]
java.util.concurrent.RejectedExecutionException: event executor terminated

I got the above error only in case when I had two config files in conf.d both of which had the input section defined .

input {
  beats {
    port => 5044
  }
}

My previous understanding was defining same configuration in multiple files logstash combines them into single file before executing it . However I think this case was executing both the inputs section that too on the same port .

Removing input section from one config file out of two resolved the error for me .

[purplesrl]

i had the same error, thanks @Ankit-Kulkarni

the problem was i had two input sections

[ph]

@purplesrl Two beats input with the same port?

[purplesrl]

@ph

I put in two .conf files with "input" sections, I wanted to unite the .conf files into one, and in that directory, there was also the separate config files (e.g. 10-input,20-filter,30-output.conf) but also total.conf