uken / fluent-plugin-elasticsearch

Apache License 2.0
890 stars 310 forks source link

Leveraging Fluentd Elastic Plugin with Elasticsearch 8.1.2 is returning ECONNREFUSED errors #962

Closed fallenreaper closed 2 years ago

fallenreaper commented 2 years ago

(check apply)

Problem

When starting a Fluentd instance using this plugin, i was finding my fluentd instance to get ECONNREFUSED errors while using the security and certificates to validate and submit data to Elastic itself. I am using the most recent version of Elasticsearch, 8.1.2, which is new as this previously worked well when using an older version of Elastic. ...

Steps to replicate

git clone https://github.com/fallenreaper/secure-EFK
docker-compose -f docker-compose-security.yaml build fluentd
docker-compose -f docker-compose-security.yaml up -d
docker-compose -f docker-compose-security.yaml logs fluentd

OR

Provide example config and message

Expected Behavior or What you need to ask

I wanted to have a solid how to on how to configure and set up this new version of elasticsearch with your existing software leveraging this plugin effectively. ...

Using Fluentd and ES plugin versions

async (1.29.0) async-http (0.54.0) async-io (1.32.1) async-pool (0.3.7) bigdecimal (default: 1.4.1) bundler (default: 1.17.2) cmath (default: 1.0.0) concurrent-ruby (1.1.9) console (1.12.0) cool.io (1.7.1) csv (default: 3.0.9) date (default: 2.0.0) dbm (default: 1.0.0) did_you_mean (1.3.0) e2mmap (default: 0.1.0) elasticsearch (7.16.1) elasticsearch-api (7.16.1) elasticsearch-transport (7.16.1) etc (default: 1.0.1) excon (0.89.0) ext_monitor (0.1.2) faraday (1.8.0) faraday-em_http (1.0.0) faraday-em_synchrony (1.0.0) faraday-excon (1.1.0) faraday-httpclient (1.0.1) faraday-net_http (1.0.1) faraday-net_http_persistent (1.2.0) faraday-patron (1.0.0) faraday-rack (1.0.0) fcntl (default: 1.0.0) fiber-local (1.0.0) fiddle (default: 1.0.0) fileutils (default: 1.1.0) fluent-plugin-elasticsearch (5.2.1, 5.1.4) fluent-plugin-secure-forward (0.4.5) fluentd (1.14.3, 1.12.4) forwardable (default: 1.2.0) gdbm (default: 2.0.0) http_parser.rb (0.8.0, 0.6.0) io-console (default: 0.4.7) ipaddr (default: 1.2.2) irb (default: 1.0.0) json (2.4.1, default: 2.1.0) logger (default: 1.3.0) matrix (default: 0.1.0) minitest (5.11.3) msgpack (1.4.2) multi_json (1.15.0) multipart-post (2.1.1) mutex_m (default: 0.1.0) net-telnet (0.2.0) nio4r (2.5.7) oj (3.10.18) openssl (default: 2.1.2) ostruct (default: 0.1.0) power_assert (1.1.3) prime (default: 0.1.0) protocol-hpack (1.4.2) protocol-http (0.21.0) protocol-http1 (0.13.2) protocol-http2 (0.14.2) proxifier (1.0.3) psych (default: 3.1.0) rake (12.3.3) rdoc (default: 6.1.2) resolv (0.2.1) resolve-hostname (0.1.0) rexml (default: 3.1.9.1) rss (default: 0.2.7) ruby2_keywords (0.0.5) scanf (default: 1.0.0) sdbm (default: 1.0.0) serverengine (2.2.4) shell (default: 0.7) sigdump (0.2.4) stringio (default: 0.0.2) strptime (0.2.5) strscan (default: 1.0.0) sync (default: 0.5.0) test-unit (3.2.9) thwait (default: 0.1.0) timers (4.3.3) tracer (default: 0.1.0) tzinfo (2.0.4) tzinfo-data (1.2021.5, 1.2021.1) webrick (default: 1.4.4) xmlrpc (0.3.0) yajl-ruby (1.4.1) zlib (default: 1.0.0)


* ES version (optional)  8.1.2
* ES template(s) (optional)
fallenreaper commented 2 years ago

Problem between keyboard and chair. Seemed that I needed the user/password properties. Was hoping to use certs instead for validation instead of passing creds, but this worked.