logstash-plugins / logstash-filter-kv

Apache License 2.0
17 stars 42 forks source link

trim_value only trimming single character at each end #59

Closed lennardk closed 6 years ago

lennardk commented 6 years ago

As of 5045f43, not only is trim_value (formerly trim) trimming only at the ends, it also it only trimming single characters at each end. Ironically, the one use mentioned in the documentation (postfix), tends to break with this.

Sample:

D30EF77C: to=<foo@example.com.org>, orig_to=<bar@example.com>, relay=mail.example.com[private/dovecot-lmtp], delay=2.2, delays=1.9/0.01/0.01/0.21, dsn=2.0.0, status=sent (250 2.0.0 <foo@example.com> A9wMBo9DwVqWBQAA3ZeTBg Saved)

After using https://github.com/whyscream/postfix-grok-patterns with this, at some point it filters "to=foo@example.com," through kv, as such:

kv {
    source       => "postfix_keyvalue_data"
    trim_value   => "<>,"
    prefix       => "postfix_"
    remove_field => [ "postfix_keyvalue_data" ]
}

The result is: postfix_to=foo@example.com> Note the trailing >.

I would expect all matching leading and trailing characters to be trimmed. If this is the intended behavior, perhaps the documentation could be clarified.

yaauie commented 6 years ago

We should definitely be able to trim all matching characters from the beginning and ends of the string, and it looks like I broke this in my giant regexp refactor that enabled field_split_pattern and value_split_pattern.

I'm coming up with a fix, but in the meantime we can use field_split_pattern => ", " to split on the comma-space sequence:

{
           "delay" => "2.2",
      "@timestamp" => 2018-05-03T22:06:42.404Z,
         "message" => "D30EF77C: to=<foo@example.com.org>, orig_to=<bar@example.com>, relay=mail.example.com[private/dovecot-lmtp], delay=2.2, delays=1.9/0.01/0.01/0.21, dsn=2.0.0, status=sent (250 2.0.0 <foo@example.com> A9wMBo9DwVqWBQAA3ZeTBg Saved)",
        "@version" => "1",
    "D30EF77C: to" => "foo@example.com.org",
          "status" => "sent (250 2.0.0 <foo@example.com> A9wMBo9DwVqWBQAA3ZeTBg Saved)",
          "delays" => "1.9/0.01/0.01/0.21",
         "orig_to" => "bar@example.com",
            "host" => "castrovel.local",
             "dsn" => "2.0.0",
           "relay" => "mail.example.com[private/dovecot-lmtp]"
}