Proposal for importing hash from LogStash event

[wblakecaldwell]

There have been other approaches to send fields from the event into InfluxDB. Most of those seem to include all fields by default, and let you exclude those you don't want. This is a different approach. You build a hash variable with LogStash filters, and send all values from that hash into InfluxDB. Key/values from the event hash are merged on top of data_points, overriding on conflict. This gives greater flexibility, allowing you to add new fields from log statements dynamically, without changing your LogStash configuration.

With this increased field flexibility, we need to add flexibility for determining data types as well. I add support for this by key prefixes for integers, floats, and booleans, with multiple prefixes allowed per type. Types declared in coerce_values override those declared by prefix.

These two changes allow users to come up with a system that lets them add key/value pairs in a log statement without updating LogStash configuration.

For example, you could configure your type prefixes like this:

data_points_type_prefixes => {
    "integer" => ["i_"]
    "float" => ["f_"]
    "boolean" => ["b_"]
}

... and use LogStash "grok" and "kv" filters to parse key-value pairs after "==>" in the following log message:

Mar 14 05:07:35 info my_server_name: Here's my log statement ==> i_userid=42; f_timing=1.42; b_authenticated=true; username="fprefect"

In this example, with the above data_points_type_prefixes, i_userid would be sent to LogStash as an integer, f_timing as a float, b_authenticated as a boolean, and username as a string. Since this plays nice with data_points, you can still also send "Here's my log statement" into InfluxDB with a separate "message" key.

[wblakecaldwell]

Ah, I might have a problem. I'm not sure if I should be merging event info into @data_points. I don't knowif @data_points is reset for each event, or if this is an instance variable for the life of the plugin. I'm now thinking it's the latter. I'll test this out some more and post back here when I have a fix or confirmation that what I'm doing is okay.

[wblakecaldwell]

Okay, I fixed the bug where I was modifying @data_points for each event. This is good to go!

[wblakecaldwell]

I just signed the CLA

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

[wblakecaldwell]

oh well ¯_(ツ)_/¯