New Plugin: filter-normalize

[webmat]

Purpose

• Facilitate the mass renaming of fields by moving the mapping out of the Logstash config to an external file
  • This will help modules normalize field names to ECS (or any other schema)
  • This may also help pipelines that need to normalize multiple inbound schemas into a common schema
• [TBC] Help with limited amount of type coercion (e.g. ensure a field is a float)

  Features

• The mapping for renames will be loaded from an external file, to keep the Logstash config uncluttered
• The plugin will offer a way to do this normalization in a few different ways:
  • Clone the event: leave the original untouched, create a normalized copy, with the specified type
  • Perform normalization on the event by renaming the fields
  • Perform normalization on the event by adding new fields, leaving original fields untouched
• If the plugin offers coercion, it will be specified directly in the mapping file

  Implementation

• Mapping file would likely be CSV (simpler to write than JSON):

source	target	conversion
type	event.type
duration	event.duration	to_float

• Missing fields would be silently ignored
  • We want to simplify the shaping of the event stream. If the user has to create tens of files for every slightly different type of events, we're not making this simpler.
• Failures to rename fields or apply conversions would:
  • add the tag _normalization_failure to the event
  • the filter would attempt to perform as much of the normalization as possible (continue trying after seeing one failure)
• Kinds of failures anticipated
  • event already has a field named like one of the destination fields (renaming "foo" => "bar.foo", but event already has field "bar.foo")
  • incorrect name for a conversion
  • conversion doesn't support the incoming type
• Plugin loading failure
  • if the CSV has incorrect headers

    Questions

    1. Does the name "normalize" sound appropriate?
  • The goal here is clearly to stay away from ECS-specific naming. This is meant to be a general purpose plugin.
  • There doesn't appear to be any other Logstash plugin named normalize.
  • Would this lead to confusion vs ElasticSearch' Normalization Token Filter? (I don't think it would)
    2. What should be the behaviour if the destination field already exists on the event?

[jakelandis]

@webmat thanks for writing this up !

+1 to purpose and features. However, I think we should support inline configuration in addition to external configuration. (mostly for better centralized config management)

Mapping file would likely be JSON

I think we should target CVS since the ECS schema is in CSV, JSON is hard to hand code and I think regardless of ECS or not, it will be row/column oriented data. Perhaps (using Kibana dot syntax):

source	target	type
type	event.type	string

For ECS, we could offer a downloadable CSV file with all the targets pre-populated with the ECS names / types. This would allow a user to look at data in Kibina and then use Excel to create the mapping file.

Does the name "normalize" sound appropriate?

I like it

What should be the behaviour if the destination field already exists on the event?

A tag on the event as _normalize_failure and leave the original event untouched. It only impacts "Perform normalization on the event by adding new fields, leaving original fields untouched" ... leaving the original untouched has some nice guarantees.

[guyboertje]

@webmat As this is a proposal and Beats and/or IngestNode might do the same, you should create a Google Doc along the lines of Beats Central Management Proposal in GDrive > Engineering [Internal] > Ingest > Proposals. The logic IIRC is to allow suggestions to be sited appropriately and further discussion on the point or someone else's suggestion are co-located. The doc is also more alive with version history. People can also add alternative ideas to the end of the doc. When consensus is reached transfer the concluded proposal here in case the community wishes to comment.

GH issues are a poor alternative.

In retrospect, my discuss issue on the math filter should have been such a doc.

[webmat]

@guyboertje I see what you mean. However I get the feeling that if I go the GDrive way, instead of a quick 1 week discussion, it could linger for much longer. The proposal here is pretty straightforward. I'll think about it, but I'm not 100% convinced yet.

[webmat]

@jakelandis Thanks for the feedback!

However, I think we should support inline configuration in addition to external configuration. (mostly for better centralized config management)

The inline configuration aspect is interesting. But if we end up supporting inline configuration, we're more or less re-implementing mutate { rename => {} }, no?

[webmat]

@jakelandis I can go with CSV, perhaps. I agree it's much simpler to write.

I don't necessarily think offering a downloadable CSV for ECS makes sense. Perhaps it does. But when I say "limited amount of coercion", what I mean is not in line with what one would find in an ECS csv file. In ECS you'll have fields that are "text" and others that are "keyword". This wouldn't fit in this mapping file. We don't want users to think they can enforce their mapping from here :-)

What I have in mind is rather a place where you can drop in all of your simple coercions:

• to_float: from string, from int, from floats that sometimes are received as 0*
• to_int: from string, from float (simple truncate)
• to_string: from int (some people prefer IDs to be keyword), from float (not sure why, but you know)

* And this 0 messing up the day's mapping if that's the first event of the day

I'd probably keep it at that for a start. I don't want to get into date/IP/geo here. We have full fledged plugins for those.

[webmat]

Also, this column would be optional (can be left empty). In most cases the field is already in the format desired.

[webmat]

Yeah the "_normalize_failure" tag makes sense on failures. I would likely try to do as many of the conversions as possible, instead of stopping at the first failure, though.