elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.18k stars 3.5k forks source link

Logstash ECS and non-ECS compliant input/filter/output #12130

Open merlixelastic opened 4 years ago

merlixelastic commented 4 years ago

As ECS is being recommended and supported by Beats and as Logstash is moving to support it, a documentation page could be added to show which modules support it now or not.

See Logstash work on ECS: https://github.com/elastic/logstash/issues/11635

yaauie commented 4 years ago

This is rather difficult to quantify.

The Elastic Common Schema provides a way to encode semantic meaning into a field's value based on the fields's name, and it is ultimately up to the user who is defining their pipelines to place extracted data in appropriate ECS-compatible fields since they are the ones who hold semantic understanding of their data.

The vast majority of plugins "work" with ECS, since they already require users to specify explicitly in which field to place which data. To be ECS-compliant, a user with knowledge of ECS simply configures the pipeline to work with the fields in a way that is ECS-complaint. For these plugins, the user is fully empowered and responsible for their own schema, whether they choose ECS or not.

A handful of plugins accept externally-structured data and blindly implicitly inject it into the root-level of the event without allowing the user to target that new data at a particular sub-field. Even these can generally be considered ECS-compliant if the user's external data is already structured in an ECS-compatable way, but for usability's sake we are working to add an optional target argument to these plugins to allow users to target the new data at a particular sub-field (where they can then manipulate it more freely to get into ECS shape if it is not already).

And a couple of our plugins, high up on the priority list to provide ECS compatiblity modes, implicly place data in fields that can't easily be coerced into ECS shape. The GeoIP filter places a handful of fields in a shape that doesn't align with ECS's geo type, and aligning it is a bit verbose. The Grok filter comes with a lot of named patterns that implictly capture to fields that don't convey the semantic meaning associated with ECS field names. Adding an ECS compatibility mode to these plugins, and the upcoming change to make that mode "on by default" in a future major release of Logstash, ensures that plugins can avoid implicitly producing events that clash with ECS without overriding user-specified explicit configuration.

The meta issue you linked includes a checklist with the current status. It is ever changing, as it is a current effort, and I don't believe that a static doc page would be the right home for a list.