fix duplicates handling, new :source & :allow_duplicates options

colinsurprenant commented 9 years ago

Fix for #3

Introduces 2 new config options:

source to specify the source field to match on, by default the message field
allow_duplicates to deduplicate or not the source field, true by default

Also I'd like to revisit the semantic of the filter_matched call in multiline. before this PR, only event matching the pattern had filter_matched called on. I think all resulting merged events should have filter_matched applied on. the match logic is really just used for bundling events together, I think the resulting events should all be considered a "match" no?

suyograo commented 9 years ago

@jsvd review please?

colinsurprenant commented 9 years ago

@jsvd thanks! let me know if you insist on splitting PR for the filter_matched logic. Also, please confirm the merge logic explanation makes sense.

colinsurprenant commented 9 years ago

this will fix and replace elasticsearch/logstash#814

jsvd commented 9 years ago

OK with the semantic change in this PR. A couple of minor comments more.

colinsurprenant commented 9 years ago

we good?

jsvd commented 9 years ago

LGTM

colinsurprenant commented 9 years ago

Good! I will :shipit:

elasticsearch-bot commented 9 years ago

Merged sucessfully into master!

logstash-plugins / logstash-filter-multiline

fix duplicates handling, new :source & :allow_duplicates options #4