Add "tags" to indicate which filters passed and which failed

[mrdavidlaing]

@dpb587, We spoke yesterday of having a @filters hash to contain information about which filters were used.

It occurs to me that we actually already store that data, but kinda mashed together into the tags field, where we put both information about successes - eg, syslog_standard, nginx, and failures - eg _grokparsefailure-nxlog_standard

I'm thinking we should categorise this existing information something like this:

@parser: { success: ['syslog_standard', 'nxlog_standard'], failures: ['_grokparsefailure-nginx_combined'] }

Danny Berger Owner dpb587 added a note 2 minutes ago I do like this idea, but this approach is more of a tagging sort of thing and logstash only supports tagging under the tags field. We'd need some sort of alternate implementation.

Alternatively, we just keep everything in tags and just do parsed-syslog_standard, parsefail-nginx_combined-grok. I think I'd like it better if it goes {rule}-{filter}(-{line/clarifier})? - then we're able to segment by all parse failures, parse failures for content types, parse failures for content types by filter.

David Laing Owner mrdavidlaing added a note just now Lets split this into a separate issue; it deserves more thought; and should't block this issue

[dpb587]

I'd like to amend my original policy suggestion with the following...

Whenever an event hits a relevant segment of code (e.g. a conditionally matched snippet for a type), the main filter (e.g. grok), should add a tag for both success and failure. The "success" tag should be the snippet/segment/logic name (e.g. nxlog_standard). The "failure" tag should be a prefix of fail/, followed by the regular tag name (e.g. nxlog_standard), followed by an optional indicator about what failed (i.e. if there's multiple groks in the same snippet, this would help identify which one).

For example, a successful log4net message sent by nxlog via syslog would have tags for syslog_standard, nxlog_standard, and log4net. If something went wrong in the nxlog parsing, it would be syslog_standard and fail/nxlog_standard.

This would allow us to use some very cheap elasticsearch filtering and aggregation using prefix queries of fail/* on the tags which we could use for automated monitoring of failures.

[mrdavidlaing]

Lets give it a try.