search

repeatedly / fluent-plugin-multi-format-parser

Multi format parser for Fluentd
108 stars 16 forks source link

fluentd-1.14.1/lib/fluent/plugin.rb:139:in `new_parser': undefined method `[]' for nil:NilClass (NoMethodError) #21

Open gitforanand opened 2 years ago

gitforanand commented 2 years ago

Fluent.Conf

<source>
      @type tail
      @label @tcpstream
      tag proxy.log
      path /fluentd/log/proxy-service-*.log
      pos_file /fluentd/log/fluentd_posfile.log.pos
      pos_file_compaction_interval 72h
    <parse>
      @type multi_format
      <pattern>
        expression /^\[(?<logtime>[^\]]*)\] (?<level>[^ ]*) (?<pod_name>[^ ]*) (?<app_name>\d*) (?<custom_dimensions>\d*) $/
        time_key logtime
        time_format %Y-%m-%d %H:%M:%S %z
      </pattern>
    </parse>
    </source>

ERROR:

/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin.rb:139:in `new_parser': undefined method `[]' for nil:NilClass (NoMethodError)
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-multi-format-parser-1.0.0/lib/fluent/plugin/parser_multi_format.rb:21:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-multi-format-parser-1.0.0/lib/fluent/plugin/parser_multi_format.rb:17:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-multi-format-parser-1.0.0/lib/fluent/plugin/parser_multi_format.rb:17:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin.rb:187:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin_helper/parser.rb:90:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin_helper/parser.rb:85:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin_helper/parser.rb:85:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin/in_tail.rb:136:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin.rb:187:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:320:in `add_source'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:161:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:155:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:155:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/engine.rb:105:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/engine.rb:80:in `run_configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/supervisor.rb:665:in `run_supervisor'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/command/fluentd.rb:352:in `'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/bin/fluentd:15:in `
repeatedly commented 2 years ago

You forget to add format regexp in <pattern>: https://github.com/repeatedly/fluent-plugin-multi-format-parser#for-v10

gitforanand commented 2 years ago

@repeatedly Works fine with that

gitforanand commented 2 years ago

Is there a way for customize for Hybrid logs. ex HALF TXT and HALF JSON 2021-11-11 02:39:11.00464 INFO proxy-service-0 ProxyLog {'di': 'XXCD', 'rc': 'cl', 'rt': 'ka', 'ts': 1636598351, 'pod': 'proxy-service-0', 'log_dt': '2021-11-11 02:39:11.464312', 'log_type': 'control_arrived', 'log_category': 'handle_device_incoming_message', 'type': 'Home'}