logstash-plugins / logstash-filter-translate

Translate filter for Logstash
Apache License 2.0
21 stars 47 forks source link

Propose to add a new setting to indicate how to handle fields that have multiple values. #66

Closed guyboertje closed 6 years ago

guyboertje commented 6 years ago

In my response to #19 I suggested this but it may get lost as that is an old issue.

I propose to add a setting that is a hint to expect multiple values on a field. Initially I put forward the name foreach but perhaps iterate_on is better.

There are at least two scenarios.

  1. There is array of strings in the field pointed to by the field setting. We want each string translated and inserted in the same position of a results array that will be added to the field pointed to by the destination setting, e.g. we have a field called "[collaborator_ids]" which holds a variable number of user ids and we want to translate these to user names and store them in a field called "[collaborator_names]"
  2. There is array of objects (as in JSON) in a field. We want to iterate on each object and translate from one field in the object to another field in the object. e.g. an array of collaboration objects in a field called "[collaborations]" where each object is a single user collaboration 'record', the record has a "[user_id]" field and we want to translate this user_id to a user name in the "[user_name]" field for each object.

Assuming a setting name of iterate_on.. To handle the first scenario above, we set iterate_on and field to the SAME value:

  iterate_on  => "[collaborator_ids]"
  field       => "[collaborator_ids]"
  destination => "[collaborator_names]"

To handle the second scenario we set iterate_on and field to be a DIFFERENT value:

  iterate_on  => "[collaborations]"
  field       => "[user_id]"
  destination => "[user_name]"

/cc @acchen97