Problem with logstash after upgrade

[Brainmoustache]

After the upgrade of Selks distro (to 4.1) logstash have trouble to restart. I do not get any alert and traffic values on the kibana dashboard.

This is the error in the log file :

[2018-04-19T08:27:00,017][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method tr' for -73.6992:Float", "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:344:inconvert_float'", "org/jruby/RubyMethod.java:120:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:inconvert'", "org/jruby/RubyArray.java:2414:in map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:inconvert'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:299:inconvert'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:252:in filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:indo_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in multi_filter'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:46:inmulti_filter'", "(eval):833:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):829:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):847:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):844:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):863:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):858:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):311:in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:398:infilter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:379:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:instart_workers'"]} [2018-04-19T08:27:00,024][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method tr' for -73.6992:Float", "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:344:inconvert_float'", "org/jruby/RubyMethod.java:120:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:inconvert'", "org/jruby/RubyArray.java:2414:in map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:inconvert'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:299:inconvert'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:252:in filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:indo_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in multi_filter'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:46:inmulti_filter'", "(eval):833:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):829:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):847:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):844:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):863:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):858:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):311:in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:398:infilter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:379:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:instart_workers'"]} [2018-04-19T08:27:00,218][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method tr' for -73.6992:Float>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:344:inconvert_float'", "org/jruby/RubyMethod.java:120:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:inconvert'", "org/jruby/RubyArray.java:2414:in map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:inconvert'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:299:inconvert'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:252:in filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:indo_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in multi_filter'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:46:inmulti_filter'", "(eval):833:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):829:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):847:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):844:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):863:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):858:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):311:in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:398:infilter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:379:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:instart_workers'"]}

[pevma]

How did you upgrade ?

[Brainmoustache]

With the upgrade script locate in the /opt/selks/Script/Setup/selks-upgrade_stamus.sh I tried the process with saying yes and no to logstash upgrade. Both upgrade didn't work.

[pevma]

That was a regular upgrade right ? aka not SELKS 3 to SELKS 4 for example ?

What is the output of dpkg -l |grep logstash ?

[Brainmoustache]

It's was a regular upgrade. Following is the result of the command dpkg -l | grep logstash:

ii logstash 1:5.6.9-1 all An extensible logging pipeline

[dgrgicevic]

Hi , to save some time in debugging, the issue is in latest logstash plugin logstash-filter-mutate

just go and

systemctl stop logstash cd /usr/share/logstash/ bin/logstash-plugin remove logstash-filter-mutate bin/logstash-plugin install --version 3.2.0 logstash-filter-mutate restart either logstash or server ..

it should work then ,

davor

On Thu, Apr 19, 2018 at 2:43 PM, Peter Manev notifications@github.com wrote:

That was a regular upgrade right ? aka not SELKS 3 to SELKS 4 for example ?

What is the output of dpkg -l |grep logstash ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/StamusNetworks/SELKS/issues/112#issuecomment-382719824, or mute the thread https://github.com/notifications/unsubscribe-auth/AbcPcgHejQOX6A6U0p6FLUlWAX5K6kVmks5tqIaIgaJpZM4TbsvZ .

-- Davor Grgicevic

[Brainmoustache]

Problem solved. Thank you.

[pevma]

Thanks for the feedback! Is this the fix - lower version install or the fix is to reinstall the plugin ?

[dgrgicevic]

it is a quick & dirty fix ... there are some inconsistency , did not have enough time to deal with this. Thank you for latest version of Scirius , excellent work ... :)

On Thu, Apr 19, 2018 at 2:57 PM, Peter Manev notifications@github.com wrote:

Thanks for the feedback! Is this the fix - lower version install or the fix is to reinstall the plugin ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/StamusNetworks/SELKS/issues/112#issuecomment-382724365, or mute the thread https://github.com/notifications/unsubscribe-auth/AbcPcuYv1MnVdNv-msWSID-EvH4VZTPNks5tqInAgaJpZM4TbsvZ .

-- Davor Grgicevic

[Brainmoustache]

The fix is to lower the version install.

[pevma]

Seems the other fix that I have found currently is to change the mutate statements form float to float_eu in /etc/logstash/conf.d/logstash.conf like so :

  if [src_ip]  {
    geoip {
      source => "src_ip" 
      target => "geoip" 
      #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" 
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float_eu" ]
    }
    if ![geoip.ip] {
      if [dest_ip]  {
        geoip {
          source => "dest_ip"
          target => "geoip"
          #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
          add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
          add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
          convert => [ "[geoip][coordinates]", "float_eu" ]
        }
      }
    }
  }
}

based on the chnages in 5.6.9 that are described here - https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-convert

Feedback is appreciated !!

[Nimdy]

wow, I have been bashing my face in for awhile now...

When will this be added to the wiki for steps to take after install?

I downloaded the ISO from the website, ran the updates, restarted.... nothing worked! Then I finally found this and everything works!

[pevma]

It is updated here now - https://github.com/StamusNetworks/SELKS/wiki/Logstash-5.6.9-breaking-upgrade