Closed michaeltravisuk closed 6 years ago
I'm experiencing the same issue:
2016-12-29 13:12:09 +0000 [warn]: section
Followed by many: 2016-12-29 13:12:36 +0000 [warn]: pattern not match: "blah blah blah..."
Using fluent-plugin-multi-format-parser-0.0.2 and td-agent 0.14.8, config (edited for public):
<source>
@type tail
path /path/to.log
pos_file /path/to/log.pos
tag insert.tag.here
format multi_format
<pattern>
format /...blah.../
time_format ...
</pattern>
<pattern>
format /...blah blah.../
time_format ...
</pattern>
</source>
The problem is v0.14's compat / parser helpers can't handle multi_format_parser's <pattern>
sub sections.
You need to use proper v0.14 parser configuration syntax like below.
<filter consul.log>
@type parser
key_name MESSAGE
<parse>
@type multi_format
<pattern>
format /^(?<time>[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}) \[(?<severity>[A-Z]*)\] (?<source>[a-z]*): (?<message>[[
^$]*)?$/
</pattern>
<pattern>
format none
</pattern>
</parse>
</filter>
The above config works well for me, thanks for clarifying.
The above config doesn't seem to be working for me on v0.14.14. Fails with:
2017-03-29 03:07:12 +0000 [error]: #0 config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="'format' parameter is required"
Adding 'format' directly under filter gets rid of error, but brings back the initial issue.
Edit: Looks like it was working up to v0.14.11. From v0.14.12+, the above error exists. Not too sure if the filter parser has been updated or a bug has been introduced.
@michaeltravisuk what was your exact working config, for both <source>
and <filter>
? Using td-agent v0.14.11 and fluent-plugin-multi-format-parser v0.1.1, I can't seem to get this working. Here's my config:
<source>
@type tail
path ...
pos_file ...
tag ...
refresh_interval 30
read_from_head true
format multi_format
</source>
<filter service.log.**>
@type parser
key_name message
<parse>
@type multi_format
<pattern>
format json
</pattern>
<pattern>
format none
</pattern>
</parse>
</filter>
With this config, I get tons of pattern not match
errors. If I remove format
from the <source>
block entirely, I get <parse> section is required
. Any thoughts?
Got this working on td-agent v0.14.16 and fluent-plugin-multi-format-parser v0.1.1 by placing the <parse/>
section in my <source/>
section. Curious as to why the README instructs users to place it in a <filter/>
instead? Anyway, here's my working conf, for anyone else who runs into this:
<source>
@type tail
path /path/to/*/service.log
pos_file /path/to/tail_pos
tag service.log.*
refresh_interval 30
read_from_head true
<parse>
@type multi_format
<pattern>
format json
</pattern>
<pattern>
format none
</pattern>
</parse>
</source>
I'm using Fluent 0.14.9 from gem source and trying to use multi_format to allow message that don't parse to fall back to being stored unformatted, but no config appears to work.
Although Fluent starts correctly, it doesn't load the patterns in format_multi. I get this output:
Dec 07 11:51:57 consul-server-a fluentd[2008]: 2016-12-07 11:51:57 +0000 [warn]: section <pattern> is not used in <filter consul.log> of parser plugin Dec 07 11:51:57 consul-server-a fluentd[2008]: 2016-12-07 11:51:57 +0000 [warn]: section <pattern> is not used in <filter consul.log> of parser plugin
Is this to do with formatting changes in newer versions of Fluent? I've tried a few different ways of writing the config but nothing has worked.
This is my config: