section <pattern> is not used

[michaeltravisuk]

I'm using Fluent 0.14.9 from gem source and trying to use multi_format to allow message that don't parse to fall back to being stored unformatted, but no config appears to work.

Although Fluent starts correctly, it doesn't load the patterns in format_multi. I get this output:

Dec 07 11:51:57 consul-server-a fluentd[2008]: 2016-12-07 11:51:57 +0000 [warn]: section <pattern> is not used in <filter consul.log> of parser plugin Dec 07 11:51:57 consul-server-a fluentd[2008]: 2016-12-07 11:51:57 +0000 [warn]: section <pattern> is not used in <filter consul.log> of parser plugin

Is this to do with formatting changes in newer versions of Fluent? I've tried a few different ways of writing the config but nothing has worked.

This is my config:

<source>
  @type systemd
  filters [{ "_SYSTEMD_UNIT": "consul.service" }]
  tag consul.log
  path /run/log/journal
</source>

<filter consul.log>
  format multi_format
  <pattern>
    format /^(?<time>[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}) \[(?<severity>[A-Z]*)\] (?<source>[a-z]*): (?<message>[[
^$]*)?$/
  </pattern>
  <pattern>
    format none
  </pattern>
  key_name MESSAGE
  @type parser
</filter>

<match consul.*>
  @type kafka
  brokers 172.17.0.3:9092
  default_topic logstash
  output_include_tag true
  output_include_time true
</match>

[davidjmemmett]

I'm experiencing the same issue: 2016-12-29 13:12:09 +0000 [warn]: section is not used in of tail plugin

Followed by many: 2016-12-29 13:12:36 +0000 [warn]: pattern not match: "blah blah blah..."

Using fluent-plugin-multi-format-parser-0.0.2 and td-agent 0.14.8, config (edited for public):

<source>
  @type tail
  path /path/to.log
  pos_file /path/to/log.pos
  tag insert.tag.here
  format multi_format

  <pattern>
    format /...blah.../
    time_format ...
  </pattern>

  <pattern>
    format /...blah blah.../
    time_format ...
  </pattern>

</source>

[repeatedly]

The problem is v0.14's compat / parser helpers can't handle multi_format_parser's <pattern> sub sections. You need to use proper v0.14 parser configuration syntax like below.

<filter consul.log>
  @type parser
  key_name MESSAGE
  <parse>
    @type multi_format
    <pattern>
      format /^(?<time>[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}) \[(?<severity>[A-Z]*)\] (?<source>[a-z]*): (?<message>[[
^$]*)?$/
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>

[michaeltravisuk]

The above config works well for me, thanks for clarifying.

[cho-m]

The above config doesn't seem to be working for me on v0.14.14. Fails with:

2017-03-29 03:07:12 +0000 [error]: #0 config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="'format' parameter is required"

Adding 'format' directly under filter gets rid of error, but brings back the initial issue.

Edit: Looks like it was working up to v0.14.11. From v0.14.12+, the above error exists. Not too sure if the filter parser has been updated or a bug has been introduced.

[baconmania]

@michaeltravisuk what was your exact working config, for both <source> and <filter>? Using td-agent v0.14.11 and fluent-plugin-multi-format-parser v0.1.1, I can't seem to get this working. Here's my config:

<source>
  @type tail
  path ...
  pos_file ...
  tag ...
  refresh_interval 30
  read_from_head true
  format multi_format
</source>

<filter service.log.**>
  @type parser
  key_name message

  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>

    <pattern>
      format none
    </pattern>
  </parse>
</filter>

With this config, I get tons of pattern not match errors. If I remove format from the <source> block entirely, I get <parse> section is required. Any thoughts?

[baconmania]

Got this working on td-agent v0.14.16 and fluent-plugin-multi-format-parser v0.1.1 by placing the <parse/> section in my <source/> section. Curious as to why the README instructs users to place it in a <filter/> instead? Anyway, here's my working conf, for anyone else who runs into this:

<source>
  @type tail
  path /path/to/*/service.log
  pos_file /path/to/tail_pos
  tag service.log.*
  refresh_interval 30
  read_from_head true

  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>

</source>