Multiline filter does not handle @metadata properly

[pascaldimassimo]

Hi,

I've been using code>@metadata</code in a configuration file for Logstash 1.5.4. My config is using metadata to represent the type of an event. I use metadata instead of type because I don't want the type field in my output (and I have multiple config in a single directory and each output will be different depending on the type). Anyway, here is a trimmed-down version of my config:

input {
    file {        
        path => "/tmp/web.log"        
        sincedb_path => "/dev/null"
        start_position => "beginning"
        add_field => { "[@metadata][type]" => "web" }
    }
}
filter {    
    multiline {                        
        pattern => "^\s"
        negate => false
        what => "previous"            
    }      
}
output {    
    if [@metadata][type] == "web" {
        stdout { codec => rubydebug { metadata => true } }        
    }
}

My test file contains 3 lines (with proper line ending):

line 1
line 2
line 3

When I execute this config, the last line is never output. If I remove the if [@metadata][type] == "web" test, that last line is output.

I looked at the code of the multiline plugin and found that metadata are not always handled properly (if I am not mistaken). In the case of the previous filter, when the event does not match the pattern, the current event is kept aside in a tmp variable and the pending events are all merged together to replace the current event. The current event, stored in tmp variable, is put in the pending array until the next event comes or a flush is triggered. Problem is, when the current event is stored in tmp, the metadata are not kept (because it is using .to_hash instead of .to_hash_with_metadata). Also, when the pending events are all merged together, they are passed to event.overwrite, which, AFAICT, is not handling metadata either. In case of a next filter, we have the same problem because of event.overwrite.

I've written a small patch that fixes the issue with my configs. I am not sure it cover all cases and if we should modify event.overwrite also. Let me know if you'd like me to make further changes to it. Or feel free to use the code for a fix if deemed necessary.

Thanks!

[purbon]

@pascaldimassimo thanks for filling this issue, while working on #22 I just tested your patch and works like charm. thanks.

[pascaldimassimo]

Perfect! Once you have a fix for #22 available, I'll test it with my config. Thanks!

[purbon]

Hi @pascaldimassimo much appreciate, you can check #23 for more details, just updated the PR a few minutes ago.

[purbon]

Fixed in https://github.com/logstash-plugins/logstash-filter-multiline/pull/23