search

fluent / fluent-plugin-windows-eventlog

Fluentd plugin to collect windows event logs
Apache License 2.0
32 stars 19 forks source link

Error when initiating fluentd daemon with plugin #82

Closed matheuspatury closed 2 years ago

matheuspatury commented 2 years ago

Describe the bug

When I start the program, it automatically exits the application.

Enviroment

OS Windows 10 PRO 19043.1348

Ruby ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x64-mingw32]

GEM

addressable (2.8.0) async (1.30.0) async-http (0.56.3) async-io (1.32.2) async-pool (0.3.8) aws-eventstream (1.1.1) aws-partitions (1.478.0) aws-sdk-core (3.117.0) aws-sdk-kms (1.44.0) aws-sdk-s3 (1.96.1) aws-sdk-sqs (1.40.0) aws-sigv4 (1.2.4) benchmark (default: 0.1.0) bigdecimal (default: 2.0.0) bundler (2.2.23, default: 2.1.4) certstore_c (0.1.7) cgi (default: 0.1.0) concurrent-ruby (1.1.9) console (1.13.1) cool.io (1.7.1 x64-mingw32) csv (default: 3.1.2) date (default: 3.0.0) dbm (default: 1.1.0) delegate (default: 0.1.0) did_you_mean (default: 1.4.0) digest-crc (0.6.4) elasticsearch (7.13.3) elasticsearch-api (7.13.3) elasticsearch-transport (7.13.3) etc (default: 1.1.0) excon (0.85.0) faraday (1.5.1) faraday-em_http (1.0.0) faraday-em_synchrony (1.0.0) faraday-excon (1.1.0) faraday-httpclient (1.0.1) faraday-net_http (1.0.1) faraday-net_http_persistent (1.2.0) faraday-patron (1.0.0) fcntl (default: 1.0.0) ffi (1.15.3 x64-mingw32) ffi-win32-extensions (1.0.4) fiber-local (1.0.0) fiddle (default: 1.0.0) fileutils (1.5.0, default: 1.4.1) fluent-config-regexp-type (1.0.0) fluent-diagtool (1.0.1) fluent-logger (0.9.0) fluent-plugin-elasticsearch (5.0.5) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.16.3) fluent-plugin-parser-winevt_xml (0.2.2) fluent-plugin-prometheus (2.0.1) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.0) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.6.0) fluent-plugin-sd-dns (0.1.0) fluent-plugin-td (1.1.0) fluent-plugin-webhdfs (1.4.0) fluent-plugin-windows-eventlog (0.8.0) fluentd (1.13.3) forwardable (default: 1.3.1) gdbm (default: 2.1.0) getoptlong (default: 0.1.0) hirb (0.7.3) http_parser.rb (0.7.0) httpclient (2.8.3) io-console (default: 0.5.6) ipaddr (default: 1.2.2) irb (default: 1.2.6) jmespath (1.4.0) json (2.5.1, default: 2.3.0) logger (default: 1.4.2) ltsv (0.1.2) matrix (default: 0.2.0) minitest (5.13.0) msgpack (1.4.2) multi_json (1.15.0) multipart-post (2.1.1) mutex_m (default: 0.1.0) net-pop (default: 0.1.0) net-smtp (default: 0.1.0) net-telnet (0.2.0) nio4r (2.5.7) nokogiri (1.11.7 x64-mingw32) observer (default: 0.1.0) oj (3.12.1) open3 (default: 0.1.0) openssl (default: 2.1.2) ostruct (default: 0.2.0) parallel (1.20.1) power_assert (1.1.7) prime (default: 0.1.1) prometheus-client (2.1.0) protocol-hpack (1.4.2) protocol-http (0.22.5) protocol-http1 (0.14.1) protocol-http2 (0.14.2) pstore (default: 0.1.0) psych (default: 3.1.0) public_suffix (4.0.6) racc (1.5.2, default: 1.4.16) rake (13.0.6, 13.0.1) rdoc (default: 6.2.1.1) readline (default: 0.0.2) reline (default: 0.1.5) rexml (default: 3.2.3.1) rss (default: 0.2.8) ruby-kafka (1.3.0) ruby-progressbar (1.11.0) ruby2_keywords (0.0.5) rubyzip (1.3.0) sdbm (default: 1.0.0) serverengine (2.2.4 x64-mingw32) sigdump (0.2.4) singleton (default: 0.1.0) stringio (default: 0.1.0) strptime (0.2.5) strscan (default: 1.0.3) td (0.16.9) td-client (1.0.8) td-logger (0.3.27) test-unit (3.3.4) timeout (default: 0.1.0) timers (4.3.3) tracer (default: 0.1.0) tzinfo (2.0.4) tzinfo-data (1.2021.1) uri (default: 0.10.0) webhdfs (0.9.0) webrick (1.7.0, default: 1.6.1) win32-api (1.10.1 universal-mingw32) win32-event (0.6.3) win32-eventlog (0.6.7) win32-ipc (0.7.0) win32-service (2.2.0) windows-api (0.4.4) windows-pr (1.2.6) winevt_c (0.9.2) xmlrpc (0.3.0) yajl-ruby (1.4.1) yaml (default: 0.1.0) zip-zip (0.3) zlib (default: 1.1.0)

Log

2021-11-19 07:27:43 -0300 [info]: parsing config file is succeeded path="C:\\opt\\td-agent\\etc\\td-agent\\td-agent.conf"
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.5'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-parser-winevt_xml' version '0.2.2'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluent-plugin-windows-eventlog' version '0.8.0'
2021-11-19 07:27:43 -0300 [info]: gem 'fluentd' version '1.13.3'
2021-11-19 07:27:44 -0300 [info]: using configuration file: <ROOT>
  <system>
    log_level info
  </system>
  <source>
    @type windows_eventlog2
    @id windows_eventlog2
    tag "winevt"
    channels application,system,security,setup
    <storage>
      @type "local"
      persistent true
      path "C:\\opt\\td-agent\\var\\post_file\\windows\\eventlog.pos"
    </storage>
  </source>
  <match winevt>
    @type file
    path "C:\\opt\\td-agent\\var\\log\\windows\\eventlog"
    append true
    <buffer>
      @type "file"
      path "C:\\opt\\td-agent\\var\\buffer\\eventlog"
      flush_mode interval
      flush_interval 5s
    </buffer>
    <format>
      @type "out_file"
    </format>
  </match>
</ROOT>
2021-11-19 07:27:44 -0300 [info]: starting fluentd-1.13.3 pid=8140 ruby="2.7.4"
2021-11-19 07:27:44 -0300 [info]: spawn command to main:  cmdline=["C:/opt/td-agent/bin/ruby.exe", "-Eascii-8bit:ascii-8bit", "C:/opt/td-agent/bin/fluentd", "-c", "C:\\opt\\td-agent\\etc\\td-agent\\td-agent.conf", "-o", "C:\\opt\\td-agent\\td-agent.log", "-x", "fluentdwinsvc", "--under-supervisor"]
2021-11-19 07:27:47 -0300 [info]: adding match pattern="winevt" type="file"
2021-11-19 07:27:47 -0300 [info]: adding source type="windows_eventlog2"
2021-11-19 07:27:48 -0300 [info]: #0 starting fluentd worker pid=6544 ppid=8140 worker=0
2021-11-19 07:27:48 -0300 [error]: #0 unexpected error error_class=TypeError error="no implicit conversion of Array into String"
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.7-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:109:in `memory'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.7-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:109:in `parse_memory'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.7-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:84:in `parse'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:260:in `bookmark_validator'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:227:in `subscription'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:197:in `block (2 levels) in refresh_subscriptions'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:181:in `retry_on_error'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:196:in `block in refresh_subscriptions'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:195:in `each'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:195:in `refresh_subscriptions'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:158:in `start'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:200:in `block in start'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:188:in `each'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:175:in `each'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:175:in `lifecycle'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:199:in `start'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:248:in `start'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:147:in `run'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:715:in `block in run_worker'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:966:in `main_process'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:706:in `run_worker'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:364:in `<top (required)>'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/bin/fluentd:23:in `load'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/bin/fluentd:23:in `<main>'
2021-11-19 07:27:48 -0300 [error]: #0 unexpected error error_class=TypeError error="no implicit conversion of Array into String"
  2021-11-19 07:27:48 -0300 [error]: #0 suppressed same stacktrace
2021-11-19 07:27:48 -0300 [info]: Worker 0 finished unexpectedly with status 1
cosmo0920 commented 2 years ago

It seems that pos file is corrupted:

2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.7-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:109:in `memory'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.7-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:109:in `parse_memory'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.7-x64-mingw32/lib/nokogiri/xml/sax/parser.rb:84:in `parse'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:260:in `bookmark_validator'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:227:in `subscription'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:197:in `block (2 levels) in refresh_subscriptions'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:181:in `retry_on_error'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:196:in `block in refresh_subscriptions'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:195:in `each'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:195:in `refresh_subscriptions'
  2021-11-19 07:27:48 -0300 [error]: #0 C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-windows-eventlog-0.8.0/lib/fluent/plugin/in_windows_eventlog2.rb:158:in `start'

Could you clean up your pos file and relaunch? Maybe, you use windows_eventlog plugin's posfile in windows_eventlog2 plugin?

matheuspatury commented 2 years ago

Maybe, you use windows_eventlog plugin's posfile in windows_eventlog2 plugin?

Yeah, pos file corrupted. Thanks