Adding a new "copy" operation

logstash-plugins / logstash-filter-mutate

Apache License 2.0

16 stars 75 forks source link

Adding a new "copy" operation #91

Open consulthys opened 7 years ago

consulthys commented 7 years ago

Here is my attempt at tackling issue #90

jsvd commented 7 years ago

this has been implemented in https://github.com/logstash-plugins/logstash-filter-mutate/pull/84

jsvd commented 7 years ago

I may have closed this too soon. it seems this feature is essentially a rename + delete:

mutate {
    copy => { "syslog" => "source" }
    remove_field => ["syslog"]
}

Do we still need this?

consulthys commented 7 years ago

The original need was expressed in #90 so if you think that covers it, I'm fine with it.

jsvd commented 7 years ago

ah I missed a few use cases:

moving a nested structured to the root of the event (the copy operation doesn't seem to support moving to root)
resetting an event's data with a portion of its nested structure (new event + moving data to root, or moving data to root + deleting everything else)

we can't close this then as these two features don't exist yet

consulthys commented 7 years ago

Weird, I thought my PR was complete... at least it supported the use cases I needed back then

jsvd commented 7 years ago

sorry, I didn't explain myself properly. I meant to say that currently the features requested in #90 aren't fulfilled in master because, although the plugin now supports copy and remove, it doesn't support copying to the root, nor clearing all event data minus a certain key.

These needs are covered in this PR, I'm just thinking if it should be refocused on providing these two features instead of implementing copy again PLUS the two features.

consulthys commented 7 years ago

Indeed. You guys should have merged this earlier ;-)

LorenKeagle commented 6 years ago

Is there a reason this is still sitting here after more than a year? Is there another feature that performs the use case discussed? I have json data from filebeat in a "json" field, and I need to move all fields up to the root event.

The reason is that ever since filebeat 6.3+, our json logs lose fields such as "host" and "input" because those are (undocumented) reserved by filebeat for some reason (why they can't just put all their beat fields in @metadata is beyond me). So we can't use the json input codec in filebeat, but we need logstash now to translate the "json" field back to root.