search

uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

Can't store ES credentials as environment variables #924

Open mattups opened 2 years ago

mattups commented 2 years ago

Problem

Hello, I'm having issues in using environment variables for ES credentials. My configs are structured like this:

td-agent.conf
|
@include elasticsearch.conf

elasticsearch.conf file is:"

@type elasticsearch
host myhost
scheme https
user "#{ENV['FLUENTD_USER']}"
password "#{ENV['FLUENTD_PASSWORD']}"

I'm running td-agent 4.2.0 fluentd 1.13.3 (12de3b5a260a174fe4a419036d6e2b2e18fe7497) on 

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

I tried to declare variable on the system with: export FLUENTD_USER=myuser export FLUENTD_PASSWORD=mypassword

But when I start up td-agent this is my log:

2021-10-22 14:48:12 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. [401] {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
2021-10-22 14:48:12 +0000 [warn]: #0 Remaining retry: 10. Retry to communicate after 32 second(s).

Seems like credentials are not interpolated. If I just put the plain text credentials in elasticsearch.conf it works just fine.

I also tried to sudo -u td-agent echo $FLUENTD_PASSWORD and it shows the correct value of the variable.

Steps to replicate

Configure td-agent to use environment variables in configs instead of plain text credentials:

user "#{ENV['FLUENTD_USER']}"
password "#{ENV['FLUENTD_PASSWORD']}"

Export the variables: 

export FLUENTD_USER=myuser
export FLUENTD_PASSWORD=mypassword

Start up td-agent service.

Expected Behavior or What you need to ask

How should this be configured to avoid plain text credentials in configs files?

Using Fluentd and ES plugin versions