search

repeatedly / fluent-plugin-multi-format-parser

Multi format parser for Fluentd
108 stars 16 forks source link

Support for format parameters inside <pattern>? #14

Closed jdeglopper closed 5 years ago

jdeglopper commented 5 years ago

Does this plugin support format parameters, such as inject_key_prefix for format json? It does not appear to work:

fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: starting fluentd-1.4.2 pid=8 ruby="2.5.5"
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-v", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-docker_metadata_tb_filter' version '0.3.2'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-elasticsearch' version '3.2.1'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-record-modifier' version '2.0.1'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluentd' version '1.4.2'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: adding filter pattern="docker.**" type="parser"
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: adding match pattern="docker.**" type="stdout"
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: adding source type="forward"
fluentd-tag | 2019-05-15 21:51:08 +0000 [warn]: fluent/log.rb:342:warn: parameter 'inject_key_prefix' in <pattern>
fluentd-tag |   format json
fluentd-tag |   inject_key_prefix jsonlog.
fluentd-tag |   reserve_data true
fluentd-tag |   reserve_time true
fluentd-tag | </pattern> is not used.
fluentd-tag | 2019-05-15 21:51:08 +0000 [warn]: fluent/log.rb:342:warn: parameter 'reserve_data' in <pattern>
fluentd-tag |   format json
fluentd-tag |   inject_key_prefix jsonlog.
fluentd-tag |   reserve_data true
fluentd-tag |   reserve_time true
fluentd-tag | </pattern> is not used.
fluentd-tag | 2019-05-15 21:51:08 +0000 [warn]: fluent/log.rb:342:warn: parameter 'reserve_time' in <pattern>
fluentd-tag |   format json
fluentd-tag |   inject_key_prefix jsonlog.
fluentd-tag |   reserve_data true
fluentd-tag |   reserve_time true
fluentd-tag | </pattern> is not used.

Minimum config file to demonstrate the error:

<system>
@log_level debug
</system>

<source>
  @type forward
  port 24224
</source>

<filter docker.**>
  @type parser
  key_name log
  <parse>
    @type multi_format
    <pattern>
      format json
      inject_key_prefix jsonlog.
      reserve_data true
      reserve_time true
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>

<match docker.**>
  @type stdout
</match>
jdeglopper commented 5 years ago

The parameters need to be configured directly under filter; this works:

<filter docker.**>
  @type parser
  key_name log
  reserve_data true
  reserve_time true
  inject_key_prefix jsonlog.
  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>
johnzheng1975 commented 3 years ago

This feature make sense, we want to identify different pattern match.