search

ip2location / logstash-filter-ip2location

IP2Location filter plugin for Logstash enables Logstash's users to add geolocation information such as country, state, city, latitude, longitude, ZIP code, time zone, ISP, domain name, connection speed, IDD code, area code, weather station code, weather station name, MNC, MCC, mobile brand, elevation and usage type by IP address.
https://www.ip2location.com
Other
20 stars 5 forks source link

Pipeline crash when use togheter with logstash-filter-ip2proxy #22

Closed rayderua closed 2 years ago

rayderua commented 2 years ago 
logstash version: 8.4.2  (also not working on 8.3.3,  8.2.3 docker/debian package)
logstash-filter-ip2location verion: 2.3.1
logstash-filter-ip2proxy verion: 2.3.1

logstash.conf:

input {
  file {
    path  => "/tmp/logstash/ipaddress.log"
    codec => "json"
  }
}

filter {
   ip2proxy {
     database => "/usr/share/IP2Location/IP2PROXY-IP-PROXYTYPE-COUNTRY.BIN"
     source => "client_ip"
     target => "ip2proxy"
   }
   ip2location {
     database => "/usr/share/IP2Location/IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE.BIN"
     source => "client_ip"
     target => "ip2location"
   }
}
output {
  file {
    path => "/tmp/logstash/output.log"  
    codec => "json_lines"
  }
}

Start add line {"client_ip": "<IP>"} to /tmp/logstash/ipaddress.log on random line pipeline crashed with error (sometimes on 1 line, some times on 15K line):

[2022-09-29T20:41:44,436][FATAL][org.logstash.Logstash    ][main] uncaught error (in thread [main]>worker1)
java.lang.NoSuchFieldError: loc
        at org.logstash.filters.IP2LocationFilter.getData(org/logstash/filters/IP2LocationFilter.java:68) ~[logstash-filter-ip2location-2.3.1.jar:?]
        at org.logstash.filters.IP2LocationFilter.handleEvent(org/logstash/filters/IP2LocationFilter.java:44) ~[logstash-filter-ip2location-2.3.1.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:427) ~[jruby.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:294) ~[jruby.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1.lib.logstash.filters.ip2location.cache_event(usr/share/logstash/vendor/bundle/jruby/$2_dot_6_dot_0/gems/logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ip2location-2.3.1/lib/logstash/filters/ip2location.rb:164) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1.lib.logstash.filters.ip2location.find(usr/share/logstash/vendor/bundle/jruby/$2_dot_6_dot_0/gems/logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ip2location-2.3.1/lib/logstash/filters/ip2location.rb:142) ~[?:?]
        at org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:171) ~[jruby.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1.lib.logstash.filters.ip2location.synchronize(usr/share/logstash/vendor/bundle/jruby/$2_dot_6_dot_0/gems/logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ip2location-2.3.1/lib/logstash/filters/ip2location.rb:170) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1.lib.logstash.filters.ip2location.find(usr/share/logstash/vendor/bundle/jruby/$2_dot_6_dot_0/gems/logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ip2location-2.3.1/lib/logstash/filters/ip2location.rb:135) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1.lib.logstash.filters.ip2location.filter(usr/share/logstash/vendor/bundle/jruby/$2_dot_6_dot_0/gems/logstash_minus_filter_minus_ip2location_minus_2_dot_3_dot_1/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ip2location-2.3.1/lib/logstash/filters/ip2location.rb:57) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178) ~[?:?]
        at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865) ~[jruby.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175) ~[?:?]
        at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(org/logstash/config/ir/compiler/FilterDelegatorExt.java:127) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset2.compute(org/logstash/generated/CompiledDataset2) ~[?:?]
        at org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:347) ~[logstash-core.jar:?]
        at org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:341) ~[logstash-core.jar:?]
        at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:87) ~[logstash-core.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:442) ~[jruby.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:306) ~[jruby.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300) ~[?:?]
        at org.jruby.RubyProc.call(org/jruby/RubyProc.java:309) ~[jruby.jar:?]
        at java.lang.Thread.run(java/lang/Thread.java:833) [?:?]

or this, if setup ip2location before ip2proxy in logstash.conf

--
[2022-09-29T20:19:23,735][FATAL][org.logstash.Logstash    ][main] uncaught error (in thread [main]>worker6)
java.lang.NoSuchFieldError: pro
        at org.logstash.filters.IP2ProxyFilter.getData(org/logstash/filters/IP2ProxyFilter.java:68) ~[logstash-filter-ip2proxy-2.3.1.jar:?]
        at org.logstash.filters.IP2ProxyFilter.handleEvent(org/logstash/filters/IP2ProxyFilter.java:44) ~[logstash-filter-ip2proxy-2.3.1.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:457) ~[jruby.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:318) ~[jruby.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_filter_minus_ip2proxy_minus_2_dot_3_dot_1.lib.logstash.filters.ip2proxy.filter(usr/share/logstash/vendor/bundle/jruby/$2_dot_6_dot_0/gems/logstash_minus_filter_minus_ip2proxy_minus_2_dot_3_dot_1/lib/logstash/filters//usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ip2proxy-2.3.1/lib/logstash/filters/ip2proxy.rb:61) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178) ~[?:?]
        at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865) ~[jruby.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(usr/share/logstash/logstash_minus_core/lib/logstash/filters//usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175) ~[?:?]
        at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(org/logstash/config/ir/compiler/FilterDelegatorExt.java:127) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset2.compute(org/logstash/generated/CompiledDataset2) ~[?:?]
        at org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:347) ~[logstash-core.jar:?]
        at org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:341) ~[logstash-core.jar:?]
        at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:87) ~[logstash-core.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:442) ~[jruby.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:306) ~[jruby.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300) ~[?:?]
        at org.jruby.RubyProc.call(org/jruby/RubyProc.java:309) ~[jruby.jar:?]
        at java.lang.Thread.run(java/lang/Thread.java:833) [?:?]
rayderua commented 2 years ago

Also tried g various configurations with use_memory_mapped / use_cache / hide_unsupported_fields configs But error still exists

ip2location commented 2 years ago

We have fixed the issue. Kindly get the latest version and try it again.

Thank you.