search

ip2location / logstash-filter-ip2location

IP2Location filter plugin for Logstash enables Logstash's users to add geolocation information such as country, state, city, latitude, longitude, ZIP code, time zone, ISP, domain name, connection speed, IDD code, area code, weather station code, weather station name, MNC, MCC, mobile brand, elevation and usage type by IP address.
https://www.ip2location.com
Other
20 stars 5 forks source link

logstash goes in error state after ip2location upgrade #14

Closed iahmad-khan closed 4 years ago

iahmad-khan commented 5 years ago

After upgrading ip2location plugin to latest version , with logtstash 6.3.0, the pipeline runs for a few hours and then randomly throws:

java.lang.Thread.run(Thread.java:748)"], :thread=>"#<Thread:0x220744cd sleep>"}
    at org.logstash.Rubyfier.deepMap(Rubyfier.java:70)
    at org.logstash.Rubyfier.lambda$initConverters$8(Rubyfier.java:98)
    at org.logstash.Rubyfier.deep(Rubyfier.java:53)
    at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_get_field(JrubyEventExtLibrary.java:83)
    at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_0_dot_2.lib.logstash.filters.ip2location.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ip2location-2.0.2/lib/logstash/filters/ip2location.rb:52)
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
    at org.jruby.runtime.Block.yield(Block.java:165)
    at org.jruby.RubyArray.each(RubyArray.java:1734)
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)
    at usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:44)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
    at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
    at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
    at org.jruby.runtime.Block.call(Block.java:124)
    at org.jruby.RubyProc.call(RubyProc.java:289)
    at org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
    at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$filter_batch$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:443)
    at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$worker_loop$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:422)
    at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384)
    at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)
    at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
    at org.jruby.runtime.Block.call(Block.java:124)
    at org.jruby.RubyProc.call(RubyProc.java:289)
    at org.jruby.RubyProc.call(RubyProc.java:246)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
    at java.lang.Thread.run(Thread.java:748)
[2018-11-26T07:51:46,684][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"", "backtrace"=>["java.util.IdentityHashMap.forEach(IdentityHashMap.java:1358)", "org.logstash.Rubyfier.deepMap(Rubyfier.java:70)", "org.logstash.Rubyfier.lambda$initConverters$8(Rubyfier.java:98)", "org.logstash.Rubyfier.deep(Rubyfier.java:53)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_get_field(JrubyEventExtLibrary.java:83)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_0_dot_2.lib.logstash.filters.ip2location.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ip2location-2.0.2/lib/logstash/filters/ip2location.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", "usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:44)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$filter_batch$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:443)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$worker_loop$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:422)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"], :thread=>"#<Thread:0x220744cd sleep>"}
Exception in thread "Ruby-0-Thread-18@[main]>worker5: :1" java.util.ConcurrentModificationException
    at java.util.IdentityHashMap.forEach(IdentityHashMap.java:1358)
    at org.logstash.Rubyfier.deepMap(Rubyfier.java:70)
    at org.logstash.Rubyfier.lambda$initConverters$8(Rubyfier.java:98)
    at org.logstash.Rubyfier.deep(Rubyfier.java:53)
    at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_get_field(JrubyEventExtLibrary.java:83)
    at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_0_dot_2.lib.logstash.filters.ip2location.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ip2location-2.0.2/lib/logstash/filters/ip2location.rb:52)
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
    at org.jruby.runtime.Block.yield(Block.java:165)
    at org.jruby.RubyArray.each(RubyArray.java:1734)
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)
    at usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:44)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
    at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
    at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
    at org.jruby.runtime.Block.call(Block.java:124)
    at org.jruby.RubyProc.call(RubyProc.java:289)
    at org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
    at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$filter_batch$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:443)
    at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$worker_loop$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:422)
    at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384)
    at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)
    at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
    at org.jruby.runtime.Block.call(Block.java:124)
    at org.jruby.RubyProc.call(RubyProc.java:289)
    at org.jruby.RubyProc.call(RubyProc.java:246)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
    at java.lang.Thread.run(Thread.java:748)
iahmad-khan commented 5 years ago

Hi,

Can you please look into this issue. We are paid customers of ip2location.

ip2location commented 5 years ago

Hi @iahmad-khan

Can you grant us access to your instance with the issue? You may send the details to support@ip2location.com to ease the investigation.

ip2location commented 4 years ago

Hi @iahmad-khan

We have updated the plugin, please try it again.