logstash-plugins / logstash-filter-grok

Grok plugin to parse unstructured (log) data into something structured.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
Apache License 2.0
122 stars 97 forks source link

Ability to auto-sort match options by frequency of match #137

Open robin13 opened 6 years ago

robin13 commented 6 years ago

A typical multi-match might look like this:

grok {
  match => { "message" => [ "%{MY_PATTERN:foo}", "%{OTHER_PATTERN:bar" ] }
}

Currently the array of matches is processed in order. Often this is good and necessary (if the patterns are a sequence of matches going from most specific to most generic), but for the case where it is simply different patterns for different strings, it would be useful if the grok filter could count the times each pattern is matched, and auto-reorder the patterns in order of the frequency they are matched and so optimise the efficiency of the grok filter.

jsvd commented 6 years ago

1) I believe first we can implement capturing of metrics per grok match 2) from there we could expose an option to automatically reorganize the patterns. However this cannot be done always and by default as it can change the semantics especially with break_on_match