Adding this plugin disrupts logstash metadata feature

welchwilmerck commented 7 years ago

I assign a value to a metadata slot in input and then use it to name the index in output. Changes behavior when this plugin is added to the filter section of the config file. Successful rest lookups result in %{[@metadata][sdlc]} being treated as a string.

input {
  jdbc {
             add_field => { "[@metadata][sdlc]" => "dev" }
...
}}

filter {
    rest {
      request => {
        url => "https://----------/%{username}"
        headers => {
         "Accept"=>"application/json"
        }
      }
     target => "person_info" 
fallback => {
"Name" => "not found" 
} }

split { 
field => "person_info"
}
 useragent { 
source => "user_agent" 
remove_field => "user_agent"
target => "ua"
}
}
output {
    stdout { codec => json_lines }
    elasticsearch {
        "index" => "logs_%{[@metadata][sdlc]}"
...
}}

Now, there's an index named logs_%{[@metadata][sdlc]} as well as separate ones for each level of sdlc.

BTW, fallback doesn't do either of what I expect. Doesn't produce a person_info with one element, Name, nor a Name field.

There are only error messages for failed lookups, which result in the correct interpretation

[2017-06-13T17:32:14,575][WARN ][logstash.filters.rest    ] rest response empty {:response=>"[]", :event=>2017-06-13T17:32:11.740Z %{host} %{message}}
[2017-06-13T17:32:14,575][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:person_info is of type = NilClass

gandalfb commented 7 years ago

Can you please post debug log? Is it reproducible with a minimal (anonymous) example? I am using this as well with @metadata but do not see this behavior.

Could you please try to add a fallback option to see if this is somehow in error handling? Is the rest service responding error or success?

welchwilmerck commented 7 years ago

Additional information.

Disruption occurs only upon SUCCESS of request.

gandalfb commented 7 years ago

Have you tried the latest version of PR #25?

Without any minimal, reproducible example (and debug log) I am not able to understand what is going on (i.e. the "..." you write, or the answer the rest sends). I hope you have the chance for an anonymous example. Maybe https://jsonplaceholder.typicode.com/ can help.

Thank you for further details.

welchwilmerck commented 7 years ago

Added more complete info to original comment. Will look into building an anonymous example.

gandalfb commented 7 years ago

Thank you.

What I gain from the 2 lines:

it seems you do not have a field username within the event, which leads to an empty rest response, which should generate an event tag '_restfailure'.
the target is not created, so that split filter is not applied, as the field not being present.
in case of an empty rest response the situation is not handled properly, as only the tag is applied, whereas it should consider fallback as well. This is thing to take care off

It would be nice if you could adjust the stdout: stdout { codec => rubydebug { metadata => true } }

And post it here as well, to see what the event looks like incl metadata.

welchwilmerck commented 7 years ago

Building plugin from source not working well, so I can't try 0.5.3 - ubuntu 14.04

There was a NoMethodError while loading logstash-filter-rest.gemspec:
undefined method `metadata=' for #<Gem::Specification name=logstash-filter-rest version=0.5.3> from
  /home/ubuntu/logstash-filter-rest/logstash-filter-rest.gemspec:26:in `block in <main>'

After commenting out metadata line:

Could not find gem 'logstash-devutils (< 2.0.0, >= 0) ruby' in the gems available on this machine.

Then

gem install logstash-devutils
ERROR:  Could not find a valid gem 'logstash-devutils' (>= 0), here is why:
          Found logstash-devutils (1.3.3), but was for platform java
^CERROR:  Interrupted

gandalfb commented 7 years ago

Yes, you need a proper dev environment with jruby.

You use logstash 5.x? Then find the rest.rb file within the logstash folder /usr/share/... or something like that and replace it with the linked file to have the latest state.

welchwilmerck commented 7 years ago

Empty rest response isn't the problem. You have it backwards. Whatever the problem with fallback, @metadata is correctly passed through.

The problem is in the code that processes a successful response. Somewhere in there, it removes @metadata.

User found - no rest filter tags and person_info (as expected), but @metadata has been removed:

{
             "referer" => "https://------",
             "is_ajax" => false,
         "person_info" => {
                 ...
    },
           "is_secure" => true,
    "annotation_docid" => "",
                  "ua" => {
                  ...
    },
    "supplement_docid" => "",
                   ...
         "ip" => #<Java::OrgPostgresqlUtil::PGobject:0x268867e5>,
           "last_name" => "K",
          "@timestamp" => 2017-06-13T18:37:24.529Z,
             "user_id" => 31,
         "search_term" => "cancer&offset=0&limit=100",
       "document_read" => "",
                "time" => 2016-08-09T17:54:46.639Z,
            "username" => "k"
}

User not found - rest filter tags and no person_info (as expected) and @metadata untouched:

{
             "referer" => "https://------",
             "is_ajax" => false,
           "is_secure" => true,
    "annotation_docid" => "",
                  "ua" => {
                  ...
    },
    "supplement_docid" => "",
                    ...
      "@metadata" => {
        "sdlc" => "dev"
    },
                  "ip" => #<Java::OrgPostgresqlUtil::PGobject:0x10cb82b4>,
           "last_name" => "M",
                "tags" => [
        [0] "_restfailure",
        [1] "_split_type_failure"
    ],
          "@timestamp" => 2017-06-13T18:37:00.530Z,
             "user_id" => 1,
         "search_term" => "",
       "document_read" => "",
                "time" => 2016-10-17T15:21:22.521Z,
            "username" => "m"
}

welchwilmerck commented 7 years ago

Same after substituting rest.rb from PR #25.

welchwilmerck commented 7 years ago

I reduced the filter to just the rest invocation and@metadata does get through for both response and no response. I'll continue to refine.

welchwilmerck commented 7 years ago

Known ES issue, apparently. split appears to be the problem: https://discuss.elastic.co/t/fields-added-using-a-ruby-filter-doesnt-survive-split-filter/57158

Thanks for the support and the debugging techniques.

lucashenning / logstash-filter-rest

Adding this plugin disrupts logstash metadata feature #26