search

lucashenning / logstash-filter-rest

REST Filter for Logstash
Other
43 stars 51 forks source link

Colon(s) in the request URL? #19

Closed jursonovicst closed 7 years ago

jursonovicst commented 7 years ago

Hi there,

Here is another one! I try to extend our logs with geoip information, so I am using the following config:

filter {
       ...
        rest {
          request => {
            url => "http://10.0.0.1:8080/GeoDirectoryServer-7.2.0/v1/ipinfo/%{ip}"
            proxy => "http://192.168.10.35:3128"
          }
          json => false
          target => "geodata"
        }
       ...
}

, where the %{ip} field holds the IP address of the client. It is working well for ipv4 addresses, but if a request comes with ipv6 then an exception is logged in logstash:

[ip] = "2003:5e:4c43:c0a9:c885:6468:a4b4:4feb"

[2017-01-30T15:07:08,807][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<LogStash::Json::ParserError: Unexpected character ('c' (code 99)): was expecting comma to separate ARRAY entries
 at [Source: [B@b85a270; line: 1, column: 78]>, "backtrace"=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:41:in `jruby_load'", "/usr/share/logstash/logstash-core/lib/logstash/json.rb:38:in `jruby_load'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-rest-0.5.1/lib/logstash/filters/rest.rb:22:in `to_object'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-rest-0.5.1/lib/logstash/filters/rest.rb:263:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:41:in `multi_filter'", "(eval):2631:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):2623:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):2663:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):2654:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):1169:in `filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:295:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:192:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:191:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:294:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:282:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:258:in `start_workers'"]}
[2017-01-30T15:07:08,833][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Json::ParserError: Unexpected character ('c' (code 99)): was expecting comma to separate ARRAY entries
 at [Source: [B@b85a270; line: 1, column: 78]>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:41:in `jruby_load'", "/usr/share/logstash/logstash-core/lib/logstash/json.rb:38:in `jruby_load'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-rest-0.5.1/lib/logstash/filters/rest.rb:22:in `to_object'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-rest-0.5.1/lib/logstash/filters/rest.rb:263:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:41:in `multi_filter'", "(eval):2631:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):2623:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):2663:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):2654:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):1169:in `filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:295:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:192:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:191:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:294:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:282:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:258:in `start_workers'"]}

debug output (bin/logstash --debug -f ...):

15:15:19.075 [[main]>worker39] DEBUG logstash.filters.rest - Parsing event fields {:sprintf_fields=>["http://10.1.1.1:8080/GeoDirectoryServer-7.2.0/v1/ipinfo/%{ip}", {}]}
15:15:19.081 [[main]>worker39] ERROR logstash.pipeline - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<LogStash::Json::ParserError: Unexpected character ('c' (code 99)): was expecting comma to separate ARRAY entries

I think the issue might be the colons in the URL (http://10.0.0.1:8080/GeoDirectoryServer-7.2.0/v1/ipinfo/2003:5e:4c43:c0a9:c885:6468:a4b4:4feb).

Of course, if I do URL Encoding with the following workarround:

      mutate { add_field => { "ip2" => "%{ip}" } }
      mutate { gsub => [ "ip2", ":", "%3a" ] }
       ...
        rest {
         ...
       }
      mutate { remove_field => [ "ip2" ] }

, then it is working well, just I find there additional lines awkward.

Would you consider checking, why these colons are triggering exceptions?

KR Tamás

gandalfb commented 7 years ago

This, indeed, is a good one :smile: Guess this will lead to a pull request @lucashenning some time "soon".

Some not thought of parsing things, hopefully to find a more correct solution.

Thanks for finding this!

gandalfb commented 7 years ago

@jursonovicst is this fixed with the new version and can be closed?

jursonovicst commented 7 years ago

@gandalfb Yes, you may close this issue, v0.5.2 is working as expected.

Thank you for your help!

T.