This plugin for Fluentd allows extracting a single key from an existing record and re-parsing it with a supplied format. A new event is then emitted, with the record modified by the now-decoded key's value.
out_burrow is designed to allow post-facto re-parsing of nested key elements.
For example, lets say your source application writes to syslog, but instead of plain string messages, it writes JSON encoded data. /var/log/syslog contains the following entry:
Jun 17 21:16:22 app1 5012162: {"event":"csrf_failure","msg":"Invalid transient key for System.","username":"System","userid":"1","ip":"192.34.93.74","method":"GET","domain":"http://timgunter.ca","path":"/dashboard/settings","tags":["csrf","failure"],"accountid":5009392,"siteid":5012162}
In td-agent.conf, you might have something like this to read this event:
<source>
type syslog
port 5140
bind 127.0.0.1
tag raw.app.vanilla.events
</source>
Unfortunately, in_syslog does not understand that the message
field is encoded with JSON, so it escapes all the data
and makes it unusable down the line. If we piped these events to a file, we would see something like this:
2014-06-17T21:16:22Z raw.app.vanilla.events.local0.err {"host":"app1","ident":"5012162","message":"{\"event\":\"csrf_failure\",\"msg\":\"Invalid transient key for System.\",\"username\":\"System\",\"userid\":\"1\",\"ip\":\"192.34.93.74\",\"method\":\"GET\",\"domain\":\"http://timgunter.ca\",\"path\":\"/dashboard/authentication\",\"tags\":[\"csrf\",\"failure\"],\"accountid\":5009392,\"siteid\":5012162}"}
Note how the message
field has been escaped. This means that when this event eventually makes its way to a file, or
another system (like elasticsearch for example), it will not be ready for consumption. That's where out_burrow
comes in.
Adding the following match
block to td-agent.conf allows us to intercept the raw syslog events and re-parse the
message field as JSON:
<match raw.app.vanilla.events.**>
type burrow
key_name message
action inplace
remove_prefix raw
format json
</match>
There are several components to this rule, but for now lets look at the output:
2014-06-17T21:16:23Z app.vanilla.events.local0.err {"host":"app1","ident":"5012162","message":{"event":"csrf_failure","msg":"Invalid transient key for System.","username":"System","userid":"1","ip":"192.34.93.74","method":"GET","domain":"http://timgunter.ca","path":"/dashboard/settings/mobilethemes","tags":["csrf","failure"],"accountid":5009392,"siteid":5012162}}
Now the JSON is no longer escaped, and can be easily parsed by both fluentd and elasticsearch.
required
This is the name of the key we want to examine and re-parse, and is required.
required
This is format that Fluentd should expect the key_name
field to be encoded with. out_burrow supports the same built-in
format as Fluent::TextParser (and in_tail):
optional
When this event is re-emitted, change its tag to this setting's value.
optional
When this event is re-emitted, remove this prefix from the source tag and use the resulting string as the new event's
tag. This setting automatically adds a trailing period .
to its value before stripping.
optional
When this event is re-emitted, prepend this prefix to the source tag and use the resulting string as the new event's tag.
This setting automatically adds a trailing period .
to its value before prepending.
One of the tag
, remove_prefix
, or add_prefix
settings is required.
remove_prefix
and add_prefix
can co-exist together.
optional
and defaults to inplace
The value of this setting determines how the new event will be constructed. There are three distinct options here:
Perform decoding 'in place'. When the key_name
field is successfully parsed, its contents will be written back to its
original key in the original record, which will then be re-emitted.
Overlay decoded data on top of original record, and re-emit. In our example above, if 'overlay' was used instead of 'inplace', the resulting record would have been:
{
"host":"app1",
"ident":"5012162",
"event":"csrf_failure",
"msg":"Invalid transient key for System.",
"username":"System",
"userid":"1",
"ip":"192.34.93.74",
"method":"GET",
"domain":"http://timgunter.ca",
"path":"/dashboard/settings",
"tags":["csrf","failure"],
"accountid":5009392,
"siteid":5012162
}
Replace the original entirely with the contents of the decoded field. In our example above, if 'replace' was used instead of 'inplace', the resulting record would have been:
{
"event":"csrf_failure",
"msg":"Invalid transient key for System.",
"username":"System",
"userid":"1",
"ip":"192.34.93.74",
"method":"GET",
"domain":"http://timgunter.ca",
"path":"/dashboard/settings",
"tags":["csrf","failure"],
"accountid":5009392,
"siteid":5012162
}
Insert the decoded data in the original record, using the data_prefix
key. In our example above, if 'prefix' was used
instead of 'inplace' with 'data_prefix' = 'data', the resulting record would have been:
{
"host":"app1",
"ident":"5012162",
"data": {
"event":"csrf_failure",
"msg":"Invalid transient key for System.",
"username":"System",
"userid":"1",
"ip":"192.34.93.74",
"method":"GET",
"domain":"http://timgunter.ca",
"path":"/dashboard/settings",
"tags":["csrf","failure"],
"accountid":5009392,
"siteid":5012162
}
}
required
if you use 'action' = 'prefix' (defaults to nil
)
The prefix used to insert the decoded data in the message.
optional
and defaults to false
Keep original source key (only valid with 'overlay' and 'replace' actions). When this is true
, the original encoded
source key is retained in the output.
optional
and defaults to false
Keep the original record's "time" key. If the original top level record contains a
optional
and defaults to time
If keep_time
is true
, this field specifies the key that contains the original records's time. The value of this key
will be copied into the new record after it has been parsed.
optional
and defaults to time
When the key_name
field's value is being parsed, look for this key and interpret it as the record's time
key.
optional
and defaults to nil
When parsing the key_name
field's value and if time_key
is set, this field denotes the format to expect the time
to be in.
git checkout -b my-new-feature
)git commit -am 'Add some feature'
)git push origin my-new-feature
)If you have a question, open an Issue.