search

janmg / logstash-filter-weblookup

logstash plugin for lookups from a configured list, redis cache or webservice (in that order)
Other
0 stars 0 forks source link

error=>"(NameError) instance variable @uri_string not defined" - Pipeline terminated {"pipeline.id"=>"nsg"} #1

Open laurentiubanica opened 3 years ago

laurentiubanica commented 3 years ago

[2021-05-18T23:42:26,274][ERROR][logstash.javapipeline ][nsg] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"nsg", :error=>"(NameError) instance variable @uri_string not defined", :exception=>Java::OrgJrubyExceptions::NameError, :backtrace=>["org.jruby.RubyKernel.remove_instance_variable(org/jruby/RubyKernel.java:2254)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.addressable_minus_2_dot_7_dot_0.lib.addressable.uri.remove_composite_values(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/addressable-2.7.0/lib/addressable/uri.rb:2525)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.addressable_minus_2_dot_7_dot_0.lib.addressable.uri.query=(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/addressable-2.7.0/lib/addressable/uri.rb:1640)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.addressable_minus_2_dot_7_dot_0.lib.addressable.uri.query_values=(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/addressable-2.7.0/lib/addressable/uri.rb:1755)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_weblookup_minus_0_dot_1_dot_3.lib.logstash.filters.weblookup.find(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-weblookup-0.1.3/lib/logstash/filters/weblookup.rb:156)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_weblookup_minus_0_dot_1_dot_3.lib.logstash.filters.weblookup.parse(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-weblookup-0.1.3/lib/logstash/filters/weblookup.rb:130)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_weblookup_minus_0_dot_1_dot_3.lib.logstash.filters.weblookup.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-weblookup-0.1.3/lib/logstash/filters/weblookup.rb:101)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1809)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:295)"], :thread=>"#"} [2021-05-18T23:42:26,279][WARN ][logstash.javapipeline ][nsg] Waiting for input plugin to close {:pipeline_id=>"nsg", :thread=>"#"} [2021-05-18T23:42:27,460][INFO ][logstash.inputs.azureblobstorage][nsg][c8e75939dc21812053267b709d61c87fc4c229be148d388bdd79e28f67aa35e9] processed 1773505 events, saving 3603 blobs and offsets to remote registry data/registry.dat [2021-05-18T23:42:27,928][INFO ][logstash.javapipeline ][nsg] Pipeline terminated {"pipeline.id"=>"nsg"}

janmg commented 3 years ago

weblookup takes a field from the pipeline and does an additional lookup and adds a json fragment. If the fragment is written to srcdst, like you would use for NSG flowlogs, the lookup will happen for src and for dst. If one field would be empty, the code will continue and Addressable is used to build the URL for the HTTP GET request. But for an empty or invalid SRC that will fail.

To continue processing a block around the code that crashed is needed to catch rather than crash.

begin doSomething(); rescue Exception => e doCleanupRoutine() end

laurentiubanica commented 3 years ago

I added a conditional statement in the filter section, to perform weblookup only if the src_ip and dst_ip fields exist. I also added a field "weblookup" with the value OK that will change to error, in case there is an issue with src_ip or dst_ip fields, so that we will be able to find and analyze the problematic events.

filter { .... mutate { add_field => { "weblookup" => "OK" } ..... }

if [src_ip] and [dst_ip] {
   weblookup {
      fields => ['src_ip','dst_ip']
      destinations => ['srcdst']
      url => "http://...?ip=<item>"
      ......
   }
   mutate {
      rename => { "srchost" => "[source][address]" }
      rename => { "srcnet" => "[source][netname]" }
      rename => { "dsthost" => "[destination][address]" }
      rename => { "dstnet" => "[destination][netname]" }
   }
}

if ![scr_ip] or ![dst_ip] {
   mutate { coerce => { "weblookup" => "error" } }
}

....... }

janmg commented 3 years ago

Great, I have now pushed 0.1.4 which should catch the exception and continue, but I don't know what would happen to the rest. The idea was to return a partial json that can be used, but I don't how it will now look like if it's empty. Your solution will catch at least that so that you know it will not be called if src or dst is empty.

laurentiubanica commented 3 years ago

Great! I'll update the filter and I'll post an update once I pick the trouble message. Thank you !!!