search

uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

Fluentd is unable to connect ElasticSearch using client certificate #615

Open prashant407 opened 5 years ago

prashant407 commented 5 years ago

Problem: We are using searchguard with elasticsearch and fluentd, I have enabled client_certificate based authentication for elasticsearch.

Configured client certificates in fluentd - so that ES can verify client connections. After providing client certificates, td-agent is unable to communicate over SSL to elasticsearch and it shows below error:

Error logs: [error]: #0 unexpected error error_class=Faraday::SSLError error="SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown (OpenSSL::SSL::SSLError)"

Followed the link: https://www.rubydoc.info/gems/fluent-plugin-elasticsearch/1.2.0

td-agent configuration:

       <match apache**>
         @type elasticsearch
         @log_level info
         include_tag_key true
         host elasticsearch.abc
         port 9200
         ca_file /etc/td-agent/certs/es-root-ca.pem
         client_cert /etc/td-agent/certs/admin.crt.pem
         client_key /etc/td-agent/certs/admin.key.pem
         scheme https
         ssl_verify true
         ssl_version TLSv1_2
         logstash_format true
         logstash_prefix prash
       </match>

But with the same certificates I am able to connect to elacticsearch with curl requests. for example: curl --insecure --cert /abc/admin.crt.pem --key /abc/admin.key.pem https://xx.xxx.xxx.xx:9200/_cat/indices?v

Above curl command gives me appropriate result.

Can you please suggest solution for this? What additional configurations are needed to enable client-cert validation with fluentd?

cosmo0920 commented 5 years ago

Could you fill the issue template? Thanks!!

prashant407 commented 5 years ago

Thanks @cosmo0920 for the response.

Using Fluentd and ES plugin versions

OS version: CentOS Linux 7 (Core) within Kubernetes td-agent version: td-agent-3.4.1-0.el7.x86_64 ES plugin: fluent-plugin-elasticsearch (3.5.1) Elasticsearch version: elasticsearch-oss-7.0.1-1.x86_64 Result of fluent-gem list: activesupport (5.2.3) addressable (2.6.0) amq-protocol (2.3.0) aws-eventstream (1.0.3) aws-partitions (1.149.0) aws-sdk-core (3.48.3) aws-sdk-kms (1.16.0) aws-sdk-s3 (1.36.0) aws-sdk-sqs (1.13.0) aws-sigv4 (1.1.0) base91 (0.0.1) bigdecimal (default: 1.3.2) bundler (1.16.6) bunny (2.14.2) concurrent-ruby (1.1.5) connection_pool (2.2.2) cool.io (1.5.4) did_you_mean (1.1.0) dig_rb (1.0.1) digest-crc (0.4.1) domain_name (0.5.20180417) elasticsearch (6.3.0) elasticsearch-api (6.3.0) elasticsearch-transport (6.3.0) excon (0.64.0) faraday (0.15.4) ffi (1.11.0) fluent-config-regexp-type (1.0.0) fluent-logger (0.8.0) fluent-plugin-amqp (0.13.0) fluent-plugin-brevity-control (0.0.2) fluent-plugin-clog (0.1.2) fluent-plugin-elasticsearch (3.5.1) fluent-plugin-forest (0.3.3) fluent-plugin-genhashvalue (1.1) fluent-plugin-kafka (0.9.4) fluent-plugin-kubernetes_metadata_filter (1.0.1) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-multiline-parser (0.1.1) fluent-plugin-postgres (0.1.0) fluent-plugin-prometheus (1.4.0) fluent-plugin-record-modifier (2.0.1) fluent-plugin-remote_syslog (1.0.0) fluent-plugin-rewrite-tag-filter (2.2.0) fluent-plugin-s3 (1.1.10) fluent-plugin-splunk-hec (1.0.1) fluent-plugin-systemd (1.0.2) fluent-plugin-td (1.0.0) fluent-plugin-td-monitoring (0.2.4) fluent-plugin-webhdfs (1.2.3) fluentd (1.4.2) hirb (0.7.3) http (0.9.8) http-cookie (1.0.3) http-form_data (1.0.3) http_parser.rb (0.6.0) httpclient (2.8.2.4) i18n (1.6.0) io-console (default: 0.4.6) ipaddress (0.8.3) jmespath (1.4.0) json (default: 2.0.4) kubeclient (1.1.4) lru_redux (1.1.0) ltsv (0.1.2) mime-types (3.2.2) mime-types-data (3.2019.0331) mini_portile2 (2.4.0) minitest (5.10.1) mixlib-cli (1.7.0) mixlib-config (2.2.4) mixlib-log (1.7.1) mixlib-shellout (2.2.7) msgpack (1.2.9) multi_json (1.13.1) multipart-post (2.1.1) murmurhash3 (0.1.6) net-http-persistent (3.0.1) net-telnet (0.1.1) netrc (0.11.0) nokogiri (1.10.3) ohai (6.20.0) oj (3.7.11) openssl (default: 2.0.9) parallel (1.17.0) pg (1.1.4) power_assert (0.4.1) prometheus-client (0.9.0) psych (default: 2.2.2) public_suffix (3.0.3) quantile (0.2.1) rake (12.3.2, 12.0.0) rdkafka (0.4.2) rdoc (default: 5.0.0) recursive-open-struct (1.0.0) remote_syslog_sender (1.2.1) rest-client (2.0.2) ruby-kafka (0.7.6) ruby-progressbar (1.10.0) rubyzip (1.2.2) serverengine (2.1.1) sigdump (0.2.4) string-scrub (0.0.5) strptime (0.2.3) syslog_protocol (0.9.2) systemd-journal (1.3.3) systemu (2.5.2) td (0.16.4) td-client (1.0.6) td-logger (0.3.27) test-unit (3.2.3) thread_safe (0.3.6) tzinfo (1.2.5) tzinfo-data (1.2019.1) unf (0.1.4) unf_ext (0.0.7.6) webhdfs (0.8.0) xmlrpc (0.2.1) yajl-ruby (1.4.1) zip-zip (0.3)

cosmo0920 commented 5 years ago

Please follow the issue template style. Don't use custom style. Thanks.

prashant407 commented 5 years ago

Problem

We are using searchguard with elasticsearch and fluentd, I have enabled client_certificate based authentication for elasticsearch. Configured client certificates in fluentd - so that ES can verify client connections. After providing client certificates, td-agent is unable to communicate over SSL to elasticsearch

Conf file being used

td-agent configuration:

<match apache**>
         @type elasticsearch
         @log_level info
         include_tag_key true
         host elasticsearch.abc
         port 9200
         ca_file /etc/td-agent/certs/es-root-ca.pem
         client_cert /etc/td-agent/certs/admin.crt.pem
         client_key /etc/td-agent/certs/admin.key.pem
         scheme https
         ssl_verify true
         ssl_version TLSv1_2
         logstash_format true
         logstash_prefix prash
       </match>

Error

[error]: #0 unexpected error error_class=Faraday::SSLError error="SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown (OpenSSL::SSL::SSLError)"

Expected Behavior or What you need to ask

After providing client certificates, td-agent is unable to communicate over SSL to elasticsearch But with the same certificates I am able to connect to elacticsearch with curl requests. for example: curl --insecure --cert /abc/admin.crt.pem --key /abc/admin.key.pem https://xx.xxx.xxx.xx:9200/_cat/indices?v

Using Fluentd and ES plugin versions

cosmo0920 commented 5 years ago

Hmm..., it's curious. Could you try this patch?

diff --git a/lib/fluent/plugin/out_elasticsearch.rb b/lib/fluent/plugin/out_elasticsearch.rb
index a943ad6..22c2b92 100644
--- a/lib/fluent/plugin/out_elasticsearch.rb
+++ b/lib/fluent/plugin/out_elasticsearch.rb
@@ -427,7 +427,7 @@ EOC
                                                                               transport_options: {
                                                                                 headers: headers,
                                                                                 request: { timeout: @request_timeout },
-                                                                                ssl: { verify: @ssl_verify, ca_file: @ca_file, version: @ssl_version }
+                                                                                ssl: { verify: @ssl_verify, ca_file: @ca_file, version: @ssl_version, certificate: @client_cert, private_key: @client_key, }
                                                                               },
                                                                               http: {
                                                                                 user: @user,
prashant407 commented 5 years ago

Thank you! can you provide me the gem so i use it directly or redirect me to the process, how to build the gem?

prashant407 commented 5 years ago

I have made the above changes in fluent-plugin-elasticsearch-3.5.1, but i am not able to build the it shows below error [root@vm-10-99-26-135 fluent-plugin-elasticsearch-3.5.1]# gem build fluent-plugin-elasticsearch.gemspec Invalid gemspec in [fluent-plugin-elasticsearch.gemspec]: No such file or directory - git ERROR: Error loading gemspec. Aborting.

could you please tell me how to build the gem file after making the above changes you have suggested?

cosmo0920 commented 5 years ago

Could you try this archived gem? (Patched ES plugin and bump up version to 3.5.1.1 [Issue confirmation only version]) fluent-plugin-elasticsearch-3.5.1.1.zip

prashant407 commented 5 years ago

Thanks @cosmo0920

I tried that gem in my Fluentd, but I got the same error. The logs are below `2019-08-01 07:01:07 +0000 [info]: Worker 0 finished unexpectedly with status 1 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-amqp' version '0.13.0' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-brevity-control' version '0.0.2' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-clog' version '0.1.2'

2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.5.1.1'

2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-forest' version '0.3.3' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-genhashvalue' version '1.1' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-kafka' version '0.9.4' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '1.0.1' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-multiline-parser' version '0.1.1' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-postgres' version '0.1.0' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.4.0' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-remote_syslog' version '1.0.0' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.10' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.0.1' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4' 2019-08-01 07:01:08 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3' 2019-08-01 07:01:08 +0000 [info]: gem 'fluentd' version '1.4.2' 2019-08-01 07:01:08 +0000 [info]: adding match pattern="apache**" type="elasticsearch" 2019-08-01 07:01:09 +0000 [error]: #0 unexpected error error_class=Faraday::SSLError error="SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown (OpenSSL::SSL::SSLError)"

cosmo0920 commented 5 years ago

Hmm...

curl --insecure --cert /abc/admin.crt.pem --key /abc/admin.key.pem https://xx.xxx.xxx.xx:9200/_cat/indices?v

Above curl command gives me appropriate result.

What is happen when remove --insecure flag in the above curl command test? I guess you use self-signed certificates, right?

If w/o --insecure curl command line causes SSL error, it seems that certificates are something wrong.

prashant407 commented 5 years ago

Our certificate is not completely self-signed certificate. We generate a root-ca certificate(signing authority) as a self-signed certificate.
openssl ca -selfsign \ -config etc/root-ca.conf \ -in ca/root-ca.csr \ -out ca/root-ca.crt \ -extensions root_ca_ext \ -batch \ -passin pass:$CA_PASS

We use this to sign my elasticsearch server certificate. So, even though elasticsearch's certificate is a signed-certificate, its signing authority is not a trusted well-known signing CA.

There is an option in curl that helps my case - --cacert FILE CA certificate to verify peer against (SSL) So by using providing the root-ca as cacert option to curl "--cacert /etc/td-agent/certs/es-root-ca.pem", we are able to reach ES and server certificate validation passes.

Is a similar parameter needed in case of the fluentd-elasticsearch plugin? I assumed the ca_file parameter did the same job - hence configured it to my root-ca.

When I run curl command from inside the fluentd pod then the result

bash-4.2# curl --cacert /etc/td-agent/certs/es-root-ca.pem --cert /etc/td-agent/certs/admin.crt.pem --key /etc/td-agent/certs/admin.key.pem https://elasticsearch.<namespace>:9200/_cat/indices?v
health status index       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   searchguard xiy1iz4nRD-CjcZF4MOz8w   1   0          6            0     25.6kb         25.6kb
prashant407 commented 5 years ago

any updates ?

prashant407 commented 5 years ago

any updates??

rlk5546 commented 4 years ago

Try this:

<match apache**>
         @type elasticsearch
         @log_level info
         include_tag_key true
         host https://elasticsearch.abc:9200
         user "username"
         password "password"
         ssl_verify false
         ssl_version TLSv1_2
         ca_file /etc/td-agent/certs/es-root-ca.pem
         logstash_format true
         logstash_prefix "prash"
       </match>
prashant407 commented 4 years ago

Hi , We are looking for certificate based authentication to elasticsearch instead of username/password. Could you please help us ?

vineel258 commented 2 years ago

@prashant407 .. any update from the last comment?