search

kiwigrid / helm-charts

Helm charts for Kubernetes curated by Kiwigrid
https://kiwigrid.github.io
MIT License
184 stars 210 forks source link

[fluentd-elasticsearch] How to use Fluentd with Elastic's chart using authentication? #392

Closed Lp-Francois closed 4 years ago

Lp-Francois commented 4 years ago

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Not sure

Version of Helm and Kubernetes: Kubernetes: 1.18.6 Helm: 3.2.1

Which chart in which version: helm-es-security: 7.8.1 (https://github.com/elastic/helm-charts/tree/7.8.1/elasticsearch/examples/security) Fluentd: 9.6.0 (This repository)

What happened: Fluentd logs: Could not communicate to Elasticsearch, resetting connection and trying again. end of file reached (EOFError)

What you expected to happen: Working with ES and authentication

How to reproduce it (as minimally and precisely as possible): 1) Install https://github.com/elastic/helm-charts/tree/7.8.1/elasticsearch/examples/security with "make secrets" and "make install". I replaced the make install command with helm install --values ./security.yml helm-es-security --version 7.8.1 ../../

2) test ES is working with port forward and curl -k -u elastic:YOUR_PASS_IN_THE_K8S_SECRET https://localhost:9200

3) install fluentd chart with these changes in values.yml:

elasticsearch:
  auth:
    enabled: true
    user: "elastic"
    password: "YOUR_PASS_IN_THE_K8S_SECRET"

  hosts: ["security-master:9200"]

4) iinspect Fluentd logs: Could not communicate to Elasticsearch, resetting connection and trying again. end of file reached (EOFError)

Anything else we need to know:

I am trying to set-up Kibana with auth. Therefore I need ES to work, so I need Fluentd to connect to ElasticSearch with authentication.

What I tried, but same error

Adding this secret in the yml:

secret:
 - name: ELASTICSEARCH_PASSWORD
   secret_name: elastic-credentials
   secret_key: password

Best,

monotek commented 4 years ago

Just put in user and password. 

elasticsearch:
  auth:
    enabled: true
    user: "user"
    password: "pass"

It's used like:

https://github.com/kiwigrid/helm-charts/blob/master/charts/fluentd-elasticsearch/templates/configmaps.yaml#L510-L513 https://github.com/kiwigrid/helm-charts/blob/master/charts/fluentd-elasticsearch/templates/daemonset.yaml#L76-L81

If you want to use a exisitng secret, by referencing it, use:

secret:
 - name: ELASTICSEARCH_PASSWORD
   secret_name: elastic-credentials
   secret_key: password

Not both.

Just check the created configmap, and env vars of the running container, to see if everything is configured right.

Lp-Francois commented 4 years ago

Hello, First thanks for your comment.

As you can read in my first post, I did add the user and password.

Indeed, I used an existing secret as well for another test, removing the auth.password from the values. (Having both should not create any problem tho).

Just check the created configmap, and env vars of the running container, to see if everything is configured right.

Already did, and not working either.

It seems like changing these values let my containers connect to ElasticSearch:

scheme: "https"
sslVerify: false

I am going to update how things are going.

Lp-Francois commented 4 years ago 
scheme: "https"
sslVerify: false

Solves the issue