search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
106 stars 3.51k forks source link

Use of @metadata in 2.0 #3694

Open untergeek opened 9 years ago

untergeek commented 9 years ago

Based on https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/218, and other similar requests, I propose that we start putting much of the metadata in inputs into the @metadata field in Logstash 2.0. This may allow for a much cleaner output for many users.

Additionally, we could have a flag (within the input block, perhaps) that, if set, puts those fields into the event, as with 1.x.

Doing this could also pave the way to using @metadata more for 2.0 plugins, which is a good thing, I think.

talevy commented 9 years ago

I agree, I think it is sometimes dangerous to have arbitrarily named fields be "keyword" fields in only specific plugins. Keeping these special fields within the @metadata namespace will allow for more flexibility in setting these special fields like type in the ES output plugin.

jordansissel commented 9 years ago

I am in favor of this change. We'll need to figure out how to handle backwards compatibility/upgrading, though. Because of this schema change, folks doing, for example, a terms agg on host will miss all events with @metadata.host.