Problem

Get the following errors as soon as I turn on the td-agent service on my RHEL8/7:

2023-06-14 04:45:26 -0500 [info]: #0 init worker0 logger path="/var/log/td-agent/td-agent.log" rotate_age=nil rotate_size=nil 2023-06-14 04:45:26 -0500 [info]: adding match pattern="td.." type="tdlog" 2023-06-14 04:45:26 -0500 [warn]: #0 [output_td] Use different plugin for secondary. Check the plugin works with primary like secondary_file primary="Fluent::Plugin::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput" 2023-06-14 04:45:26 -0500 [info]: adding match pattern="debug." type="stdout" 2023-06-14 04:45:26 -0500 [info]: adding match pattern="system." type="elasticsearch" 2023-06-14 04:45:26 -0500 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)" 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:324:in rescue in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:285:inperform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:36:in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/client.rb:176:inperform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-8.7.0/lib/elasticsearch.rb:71:in method_missing' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-api-8.7.0/lib/elasticsearch/api/actions/info.rb:41:ininfo' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:498:in detect_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:489:inblock in handle_last_seen_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/elasticsearch_index_template.rb:56:in retry_operate' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:486:inhandle_last_seen_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:338:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:inconfigure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in add_match' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:inblock in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in each' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:inconfigure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:inconfigure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in run_configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:616:inblock in run_worker' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:962:in main_process' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:608:inrun_worker' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:372:in <top (required)>' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:inrequire' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in<top (required)>' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in load' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in

' 2023-06-14 04:45:26 -0500 [error]: Worker 0 exited unexpectedly with status 1

Steps to replicate

Syslog Conf snippet from td-agent.conf

Get logs from SYSLOG

@type syslog port 5140 tag system

<match system.**> @type elasticsearch host localhost port 9200 logstash_format true

Expected Behavior or What you need to ask

Integrated Elasticsearch with Kibana. Expecting to filter and analyze the logs on Kibana dashboard

Using Fluentd and ES plugin versions

RHEL8/7 VMWare VM Fluentd v1.0 td-agent 4.5.0 fluentd 1.16.1 (0a6d706a9cee5882d751b2cc6169696709df0134) fluent-plugin-elasticsearch-5.3.0

td-agent-gem list

LOCAL GEMS

addressable (2.8.4) async (1.31.0) async-http (0.60.1) async-io (1.34.3) async-pool (0.4.0) aws-eventstream (1.2.0) aws-partitions (1.752.0) aws-sdk-core (3.171.0) aws-sdk-kms (1.63.0) aws-sdk-s3 (1.121.0) aws-sdk-sqs (1.53.0) aws-sigv4 (1.5.2) benchmark (default: 0.1.0) bigdecimal (default: 2.0.0) bindata (2.4.15) bundler (2.3.26, default: 2.1.4) cgi (default: 0.1.0.2) cmetrics (0.3.3) concurrent-ruby (1.2.2) console (1.16.2) cool.io (1.7.1) csv (default: 3.1.2) date (default: 3.0.3) delegate (default: 0.1.0) did_you_mean (default: 1.4.0) digest-crc (0.6.4) digest-murmurhash (1.1.1) elastic-transport (8.2.1) elasticsearch (8.7.0) elasticsearch-api (8.7.0) etc (default: 1.1.0) excon (0.99.0) faraday (2.7.4) faraday-excon (2.1.0) faraday-net_http (3.0.2) faraday_middleware-aws-sigv4 (1.0.1) fcntl (default: 1.0.0) ffi (1.15.5) fiber-local (1.0.0) fiddle (default: 1.0.0) fileutils (1.7.1, default: 1.4.1) fluent-config-regexp-type (1.0.0) fluent-diagtool (1.0.1) fluent-logger (0.9.0) fluent-plugin-calyptia-monitoring (0.1.3) fluent-plugin-elasticsearch (5.3.0) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.19.0) fluent-plugin-metrics-cmetrics (0.1.2) fluent-plugin-opensearch (1.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-sd-dns (0.1.0) fluent-plugin-systemd (1.0.5) fluent-plugin-td (1.2.0) fluent-plugin-utmpx (0.5.0) fluent-plugin-webhdfs (1.5.0) fluentd (1.16.1) forwardable (default: 1.3.1) getoptlong (default: 0.1.0) hirb (0.7.3) http_parser.rb (0.8.0) httpclient (2.8.3) io-console (default: 0.5.6) ipaddr (default: 1.2.2) irb (default: 1.2.6) jmespath (1.6.2) json (2.6.3, default: 2.3.0) linux-utmpx (0.3.0) logger (default: 1.4.2) ltsv (0.1.2) matrix (default: 0.2.0) mini_portile2 (2.8.1) minitest (5.13.0) msgpack (1.7.0) multi_json (1.15.0) mutex_m (default: 0.1.0) net-pop (default: 0.1.0) net-smtp (default: 0.1.0) net-telnet (0.2.0) nio4r (2.5.9) observer (default: 0.1.0) oj (3.14.3) open3 (default: 0.1.0) opensearch-api (2.2.0) opensearch-ruby (2.1.0) opensearch-transport (2.1.0) openssl (default: 2.1.4) ostruct (default: 0.2.0) parallel (1.20.1) power_assert (1.1.7) prime (default: 0.1.1) prometheus-client (2.1.0) protocol-hpack (1.4.2) protocol-http (0.24.1) protocol-http1 (0.15.0) protocol-http2 (0.15.1) pstore (default: 0.1.0) psych (default: 3.1.0) public_suffix (5.0.1) racc (default: 1.4.16) rake (13.0.6, 13.0.1) rdkafka (0.11.1) rdoc (default: 6.2.1.1) readline (default: 0.0.2) readline-ext (default: 0.1.0) reline (default: 0.1.5) rexml (3.2.5, default: 3.2.3.1) rss (default: 0.2.8) ruby-kafka (1.5.0) ruby-progressbar (1.13.0) ruby2_keywords (0.0.5) rubyzip (1.3.0) sdbm (default: 1.0.0) serverengine (2.3.2) sigdump (0.2.4) singleton (default: 0.1.0) stringio (default: 0.1.0) strptime (0.2.5) strscan (default: 1.0.3) systemd-journal (1.4.2) td (0.17.1) td-client (1.0.8) td-logger (0.3.28) test-unit (3.3.4) timeout (default: 0.1.0) timers (4.3.5) tracer (default: 0.1.0) traces (0.9.1) tzinfo (2.0.6) tzinfo-data (1.2023.3) uri (default: 0.10.0.2) webhdfs (0.10.2) webrick (1.8.1, default: 1.6.1) xmlrpc (0.3.0) yajl-ruby (1.4.3) yaml (default: 0.1.0) zip-zip (0.3) zlib (default: 1.1.0) [root@nclpvnlapp10032 td-agent]#

uken / fluent-plugin-elasticsearch

Not able to send syslogs to Elasticsearch using fluent-plugin-elasticsearch #1020