search

kiwigrid / helm-charts

Helm charts for Kubernetes curated by Kiwigrid
https://kiwigrid.github.io
MIT License
186 stars 209 forks source link

Fluentd-elasticsearch : ES connection - error_class=Elasticsearch::Transport::Transport::Errors::Unauthorized error="[401] #362

Closed plaformsre closed 4 years ago

plaformsre commented 4 years ago

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report

Version of Helm and Kubernetes: Helm 3.2, K8S 1.14

Which chart in which version: Latest fluentd-elasticsearch 9.3.1

What happened: Cannot authenticate to ES K8S service endpoint where username and password are correct, same with the ES endpoint (version 7.4.2). RBAC/Security enabled on ES. We have TLS enabled with self signed certificates. Tls verify is se to false, tls protocol set to 1_2, user is an admin user in ES.

Is there a possibility to add CA certs to the chart and how? We use ECK and certs are stored as K8S secrets

What you expected to happen: FluentD to authenticate against the ES K8S service. Fluentd and ES reside in the same namespace.

How to reproduce it (as minimally and precisely as possible): Provided username and password under auth section, also as env variable, also tried a K8S secret, did not help. We did not provide any certificates for fluentd.

Anything else we need to know:

Error message from a running fluentd-elasticsearch pod. No issues with the daemonset.

2020-06-03 17:54:46 +0000 [warn]: [elasticsearch] Could not connect Elasticsearch or obtain version. Assuming Elasticsearch 5. 2020-06-03 17:54:48 +0000 [warn]: [elasticsearch] failed to write data into buffer by buffer overflow action=:block 2020-06-03 17:54:49 +0000 [warn]: [elasticsearch] failed to write data into buffer by buffer overflow action=:block 2020-06-03 17:54:49 +0000 [warn]: [elasticsearch] failed to write data into buffer by buffer overflow action=:block 2020-06-03 17:54:49 +0000 [warn]: [elasticsearch] failed to write data into buffer by buffer overflow action=:block 2020-06-03 17:54:49 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2020-06-03 17:54:50 +0000 chunk="5a7314c2e821f794e77825c72a0df07a" error_class=Elasticsearch::Transport::Transport::Errors::Unauthorized error="[401] " 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/base.rb:205:in __raise_transport_error' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/base.rb:323:inperform_request' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/http/faraday.rb:20:in perform_request' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/client.rb:131:inperform_request' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-api-6.1.0/lib/elasticsearch/api/actions/ping.rb:20:in ping' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:344:inclient' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:631:in rescue in send_bulk' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:618:insend_bulk' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:523:in block in write' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:522:ineach' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:522:in write' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.2.4/lib/fluent/plugin/output.rb:1110:intry_flush' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.2.4/lib/fluent/plugin/output.rb:1389:in flush_thread_run' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.2.4/lib/fluent/plugin/output.rb:444:inblock (2 levels) in start' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.2.4/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create' 2020-06-03 17:54:49 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2020-06-03 17:54:50 +0000 chunk="5a7314c1cb03f4402a40f3be4cd8b8f5" error_class=Elasticsearch::Transport::Transport::Errors::Unauthorized error="[401] " 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/base.rb:205:in__raise_transport_error' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/base.rb:323:in perform_request' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/http/faraday.rb:20:inperform_request' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/client.rb:131:in perform_request' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/elasticsearch-api-6.1.0/lib/elasticsearch/api/actions/ping.rb:20:inping' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:344:in client' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:631:inrescue in send_bulk' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:618:in send_bulk' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:523:inblock in write' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:522:in each' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.11.11/lib/fluent/plugin/out_elasticsearch.rb:522:inwrite' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.2.4/lib/fluent/plugin/output.rb:1110:in try_flush' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.2.4/lib/fluent/plugin/output.rb:1389:inflush_thread_run' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.2.4/lib/fluent/plugin/output.rb:444:in block (2 levels) in start' 2020-06-03 17:54:49 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.2.4/lib/fluent/plugin_helper/thread.rb:78:inblock in thread_create' 2020-06-03 17:54:50 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=1 next_retry_seconds=2020-06-03 17:54:50 +0000 chunk="5a7314c1cb03f4402a40f3be4cd8b8f5" error_class=Elasticsearch::Transport::Transport::Errors::Unauthorized error="[401] " 2020-06-03 17:54:50 +0000 [warn]: suppressed same stacktrace 2020-06-03 17:54:50 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=2 next_retry_seconds=2020-06-03 17:54:52 +0000 chunk="5a7314c2e821f794e77825c72a0df07a" error_class=Elasticsearch::Transport::Transport::Errors::Unauthorized error="[401] " 2020-06-03 17:54:50 +0000 [warn]: suppressed same stacktrace 2020-06-03 17:54:53 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=3 next_retry_seconds=2020-06-03 17:54:53 +0000 chunk="5a7314c2e821f794e77825c72a0df07a" error_class=Elasticsearch::Transport::Transport::Errors::Unauthorized error="[401] " 2020-06-03 17:54:53 +0000 [warn]: suppressed same stacktrace 2020-06-03 17:54:53 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-06-03 17:55:01 +0000 chunk="5a7314c1cb03f4402a40f3be4cd8b8f5" error_class=Elasticsearch::Transport::Transport::Errors::Unauthorized error="[401] " 2020-06-03 17:54:53 +0000 [warn]: suppressed same stacktrace 2020-06-03 17:54:54 +0000 [warn]: [elasticsearch] failed to write data into buffer by buffer overflow action=:block

plaformsre commented 4 years ago

To be closed, issue was a user with insufficient permissions.