search

logstash-plugins / logstash-input-s3

Apache License 2.0
57 stars 150 forks source link

Implement ECS-Compatibility Mode #196

Closed yaauie closed 3 years ago

yaauie commented 4 years ago

This is a stub issue, and needs to be fleshed out with details specific to this plugin.

As a part of the effort to make plugins able to run in an ECS-Compatible manner by default in an upcoming release of Logstash, this plugin needs to either implement an ECS-Compatibility mode or certify that it does not implicitly use fields that conflict with ECS.

kaisecheng commented 3 years ago

One of the acceptance criteria is retargeting cloudfront_version, cloudfront_fields from root level to ECS compatible fields, but none of ECS fields fit these two.

cloudfront_version is similar to event version. One log has many lines. Each line is an event. Events could be in the same version. cloudfront_fields is more like an event metadata showing the columns name. Can I retarget them to event.version and event.metadata.description ?

What do you think @yaauie ?

kaisecheng commented 3 years ago

just checked beat-input ecs, maybe we can map as the following

Legacy ECS
cloudfront_fields [@metadata][s3][cloudfront][fields]
cloudfront_version [@metadata][s3][cloudfront][version]
yaauie commented 3 years ago

I am +1 to adding both the cloudfront version and fields captures into the event's @metadata, and allowing users to pull them into the main event if and when they have a need to do so.