search

uken / fluent-plugin-elasticsearch

Apache License 2.0
892 stars 309 forks source link

Not able to send syslogs to Elasticsearch using fluent-plugin-elasticsearch #1020

Closed valleysaint closed 1 year ago

valleysaint commented 1 year ago

Problem

Get the following errors as soon as I turn on the td-agent service on my RHEL8/7:

2023-06-14 04:45:26 -0500 [info]: #0 init worker0 logger path="/var/log/td-agent/td-agent.log" rotate_age=nil rotate_size=nil 2023-06-14 04:45:26 -0500 [info]: adding match pattern="td.." type="tdlog" 2023-06-14 04:45:26 -0500 [warn]: #0 [output_td] Use different plugin for secondary. Check the plugin works with primary like secondary_file primary="Fluent::Plugin::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput" 2023-06-14 04:45:26 -0500 [info]: adding match pattern="debug." type="stdout" 2023-06-14 04:45:26 -0500 [info]: adding match pattern="system." type="elasticsearch" 2023-06-14 04:45:26 -0500 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)" 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:324:in rescue in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:285:inperform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:36:in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/client.rb:176:inperform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-8.7.0/lib/elasticsearch.rb:71:in method_missing' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-api-8.7.0/lib/elasticsearch/api/actions/info.rb:41:ininfo' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:498:in detect_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:489:inblock in handle_last_seen_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/elasticsearch_index_template.rb:56:in retry_operate' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:486:inhandle_last_seen_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:338:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:inconfigure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in add_match' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:inblock in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in each' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:inconfigure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:inconfigure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in run_configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:616:inblock in run_worker' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:962:in main_process' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:608:inrun_worker' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:372:in <top (required)>' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:inrequire' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in<top (required)>' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in load' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in

' 2023-06-14 04:45:26 -0500 [error]: Worker 0 exited unexpectedly with status 1

Steps to replicate

Syslog Conf snippet from td-agent.conf

Get logs from SYSLOG

@type syslog port 5140 tag system

<match system.**> @type elasticsearch host localhost port 9200 logstash_format true

Expected Behavior or What you need to ask

Integrated Elasticsearch with Kibana. Expecting to filter and analyze the logs on Kibana dashboard

Using Fluentd and ES plugin versions

RHEL8/7 VMWare VM Fluentd v1.0 td-agent 4.5.0 fluentd 1.16.1 (0a6d706a9cee5882d751b2cc6169696709df0134) fluent-plugin-elasticsearch-5.3.0

td-agent-gem list

LOCAL GEMS

addressable (2.8.4) async (1.31.0) async-http (0.60.1) async-io (1.34.3) async-pool (0.4.0) aws-eventstream (1.2.0) aws-partitions (1.752.0) aws-sdk-core (3.171.0) aws-sdk-kms (1.63.0) aws-sdk-s3 (1.121.0) aws-sdk-sqs (1.53.0) aws-sigv4 (1.5.2) benchmark (default: 0.1.0) bigdecimal (default: 2.0.0) bindata (2.4.15) bundler (2.3.26, default: 2.1.4) cgi (default: 0.1.0.2) cmetrics (0.3.3) concurrent-ruby (1.2.2) console (1.16.2) cool.io (1.7.1) csv (default: 3.1.2) date (default: 3.0.3) delegate (default: 0.1.0) did_you_mean (default: 1.4.0) digest-crc (0.6.4) digest-murmurhash (1.1.1) elastic-transport (8.2.1) elasticsearch (8.7.0) elasticsearch-api (8.7.0) etc (default: 1.1.0) excon (0.99.0) faraday (2.7.4) faraday-excon (2.1.0) faraday-net_http (3.0.2) faraday_middleware-aws-sigv4 (1.0.1) fcntl (default: 1.0.0) ffi (1.15.5) fiber-local (1.0.0) fiddle (default: 1.0.0) fileutils (1.7.1, default: 1.4.1) fluent-config-regexp-type (1.0.0) fluent-diagtool (1.0.1) fluent-logger (0.9.0) fluent-plugin-calyptia-monitoring (0.1.3) fluent-plugin-elasticsearch (5.3.0) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.19.0) fluent-plugin-metrics-cmetrics (0.1.2) fluent-plugin-opensearch (1.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-sd-dns (0.1.0) fluent-plugin-systemd (1.0.5) fluent-plugin-td (1.2.0) fluent-plugin-utmpx (0.5.0) fluent-plugin-webhdfs (1.5.0) fluentd (1.16.1) forwardable (default: 1.3.1) getoptlong (default: 0.1.0) hirb (0.7.3) http_parser.rb (0.8.0) httpclient (2.8.3) io-console (default: 0.5.6) ipaddr (default: 1.2.2) irb (default: 1.2.6) jmespath (1.6.2) json (2.6.3, default: 2.3.0) linux-utmpx (0.3.0) logger (default: 1.4.2) ltsv (0.1.2) matrix (default: 0.2.0) mini_portile2 (2.8.1) minitest (5.13.0) msgpack (1.7.0) multi_json (1.15.0) mutex_m (default: 0.1.0) net-pop (default: 0.1.0) net-smtp (default: 0.1.0) net-telnet (0.2.0) nio4r (2.5.9) observer (default: 0.1.0) oj (3.14.3) open3 (default: 0.1.0) opensearch-api (2.2.0) opensearch-ruby (2.1.0) opensearch-transport (2.1.0) openssl (default: 2.1.4) ostruct (default: 0.2.0) parallel (1.20.1) power_assert (1.1.7) prime (default: 0.1.1) prometheus-client (2.1.0) protocol-hpack (1.4.2) protocol-http (0.24.1) protocol-http1 (0.15.0) protocol-http2 (0.15.1) pstore (default: 0.1.0) psych (default: 3.1.0) public_suffix (5.0.1) racc (default: 1.4.16) rake (13.0.6, 13.0.1) rdkafka (0.11.1) rdoc (default: 6.2.1.1) readline (default: 0.0.2) readline-ext (default: 0.1.0) reline (default: 0.1.5) rexml (3.2.5, default: 3.2.3.1) rss (default: 0.2.8) ruby-kafka (1.5.0) ruby-progressbar (1.13.0) ruby2_keywords (0.0.5) rubyzip (1.3.0) sdbm (default: 1.0.0) serverengine (2.3.2) sigdump (0.2.4) singleton (default: 0.1.0) stringio (default: 0.1.0) strptime (0.2.5) strscan (default: 1.0.3) systemd-journal (1.4.2) td (0.17.1) td-client (1.0.8) td-logger (0.3.28) test-unit (3.3.4) timeout (default: 0.1.0) timers (4.3.5) tracer (default: 0.1.0) traces (0.9.1) tzinfo (2.0.6) tzinfo-data (1.2023.3) uri (default: 0.10.0.2) webhdfs (0.10.2) webrick (1.8.1, default: 1.6.1) xmlrpc (0.3.0) yajl-ruby (1.4.3) yaml (default: 0.1.0) zip-zip (0.3) zlib (default: 1.1.0) [root@nclpvnlapp10032 td-agent]#

valleysaint commented 1 year ago

curl -i 127.0.0.1:9200

curl: (52) Empty reply from server

https://stackoverflow.com/questions/35921195/curl-52-empty-reply-from-server-timeout-when-querying-elastiscsearch

I disabled the xpack security in ES configuration and it worked.

In /etc/elasticsearch/elasticsearch.yml2, changed

xpack.security.enabled: false

Restarted ES service and the errors in td-agent were gone.