KV filter dropping existing fields in target object v2.0.2 -> v2.0.3

[berglh]

I didn't confirm this, but my suspicion is that the changes in the following commit seems to override any fields in the target object for the KV filter: refactor field refereces to not rely on in-place mutability. @colinsurprenant @ph

In my particular case, I add a few fields to an object in a grok filter, then I run the KV filter targeting that object to add the KV fields to the object. In version 2.0.2, the plugin does not drop the existing fields, in version 2.0.3, the plugin drops any fields in the target object before adding the KV pairs as fields.

• Version: Logstash 2.2.1 and above, logstash-filter-kv 2.0.3 and above
• Operating System: Oracle Enterprise Linux 7.2
• Config File: See below
• Sample Data: Apply to any log data, these filters just add the required example
• Steps to Reproduce:
  1. Create a JSON object with some fields and values in a grok pattern or using mutate i.e.

     mutate {
     add_field => [
     "[object][field1]", "value1",
     "[object][field2]", "value2"
     ]
     }

  2. Now use that [object] as a target for the KV filter:

     mutate {
     add_field => {
     "[kv-string]" => "&Packet-Type=Access-Accept&Session-Timeout=2573737&qnsService=STAFF_USER&Class=qnsService=STAFF_USER&Framed-IP-Address=172.18.100.01"
     } 
     }
     kv {
     source => "[kv-string]"
     target => "object"
     field_split => "\&"
     value_split => "="
     }

     What happens is the new key values exist in the [object], but existing fields [field1] and [field2] in the object are dropped.

[darkmoon03]

This happens in 5.4.1 as well.

[hrak]

This is still happening in kv filter version logstash-filter-kv-4.1.2

I worked around it for now with something like:

  kv {
    source => "[auditd][log][kv]"
    field_split => "\s+"
    value_split => "="
    target => "temp_kv"
    remove_field => "[auditd][log][kv]"
  }

  if [temp_kv] {
    mutate {
      merge => {"[auditd][log]" => "temp_kv"}
      remove_field => "temp_kv"
    }
  }

[colinsurprenant]

submitted PR #69 to fix this regression. This regression has made it through 2 major versions of the plugin but nonetheless I think it makes sense to not loose existing fields in the target if it exists.

[widhalmt]

I do have the exact same problem.

[beanaroo]

I can confirm that this issue is still present:

    # Invocation Statistics
    grok {
        match => { "message" => "(?:START|END|REPORT) RequestId: %{UUID:[lambda][RequestId]}%{SPACE}%{GREEDYDATA:[@metadata][lambda_report]}" }
        tag_on_failure => []
    }

    kv {
      source => "[@metadata][lambda_report]"
      target => "lambda"
      value_split => ":"
      field_split_pattern => "\t"
      remove_char_key => " "
    }

[maltewhiite]

Having this exact same problem. This really sucks. The fields don't overlap, but it simply destroys the object, before putting in the new values. Please fix this. Or at least add to the Documentation that using the KV filter will destroy the object, then create a new one, with the KV filter values. I assumed I could make a cool object with mutate, kv, etc. But KV just destroys it. Wasted 4 hours today ripping my hair out, before I stumbled on this issue, which explained what was happening.