Access environment variables from elasticsearch.yml

[ghost]

Great work, the image works brilliantly! I'm facing one issue, that when I customize elasticsearch.yml with my own files, the environment variable substitutions don't work, ES returns an exception like "Could not resolve placeholder 'CLUSTER_NAME'".

I've found this relevant SO post with a solution, but I'm not entirely sure how to apply this as a generic solution: https://stackoverflow.com/questions/24964018/elasticsearch-cant-resolve-environment-variables-in-elasticsearch-yml

Any ideas?

[spujadas]

As far as I know there is no generic solution: as the services (ES, LS and Kibana) do not use the environment variables that are passed at runtime (as mentioned in the link you provided), the only way to achieve what you describe are through hacks such as:

• Using sed or awk to replace specific variables in the services' start-up files with the value passed at runtime (e.g. https://github.com/spujadas/elk-docker/blob/master/start.sh#L69-L72 sets ES's ES_JAVA_OPTS env var by changing ES's /etc/default/elasticsearch file).

• Customising the /etc/default/elasticsearch to pass all the env vars you need... which probably isn't a very dynamic way to set variables.

Having said that, if you only need the CLUSTER_NAME variable to be available in your elasticsearch.yml, that would be a fairly generic variable that I could expose in the default image.

[ghost]

Hmm, I thought about the first option indeed (patching start.sh) but I'm reluctant to do it because I might lose future features from your future image releases :)

Indeed there's really a limited number of variables I'm interested to pass, basically just enough to be able to run master and slave cluster members from the same image. So for this I guess I would only need those:

cluster.name: "${CLUSTER_NAME}"
network.publish_host: "${PUBLISH_HOST}"
node.name: "${PUBLISH_HOST}"
discovery.zen.ping.unicast.hosts: [ $CLUSTER_MEMBERS ]

I don't have much experience with clusters but it looks like this should be enough. Does it make sense?

Thanks!

[spujadas]

Yep, makes perfect sense.

What I could do, and that might be the most generic way to do this, is add a hook in start.sh which looks for a file named (for instance) /opt/elasticsearch/env.sh, and if it exists, then it sources it. Then that file could include all the necessary sed/awk wizardry to process whatever environment variables are needed, and could easily be bind-mounted at runtime. And this would work with future releases of the image.

[ghost]

Yes, sounds like a good option. In fact, I'm thinking that it should be doable to emulate Elastic's runtime variable substitution by pure sed/awk manipulations in a generic way (i.e. for any ${VARIABLE} placeholder)... So there might still be a generic solution possible. But the quickest is indeed probably to source a pre-flight script at startup that we can easily override.

[spujadas]

OK, just added the feature to the latest (tag: 550) version of the image. Give it a shot (see http://elk-docker.readthedocs.io/#pre-hooks) and let me know how it goes.

[ghost]

Hey, looks great, thanks, it really seems to fit the need! I cannot test it though because I'm actually using the elkx (w/ Xpack) image, which doesn't seem to have the change yet...

[spujadas]

Yep, had forgotten to update the elkx image: build is currently in progress.

[tedder]

woot, this is coming in handy for me- for publishing the proper IP address of a containerized Logstash instance.

[ghost]

Just back from vacation and could test it out! Actually for some reason I've had mixed results:

• Works fine for logstash - I'm parameterizing the elastic password as follows:

    elasticsearch {
      hosts => [ "localhost:9200" ]
      user => elastic
      password => "${ES_PASSWORD:changeme}"
    }

• But for Elastic I'm getting the exception at elastic startup, which is pretty explicit: Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'HOST_IP'

The pre-hook script gets properly executed as I can see the generated /etc/defaults/elasticsearch file in the container and it seems properly formed:

... (default file content)
#MAX_MAP_COUNT=262144
CLUSTER_NAME="RSD_ELK_PRD"
ES_NODE_NAME="7f0c7d061956"
HOST_IP="172.21.224.78"

And my elasticsearch.yml is as follows:

network.host: 0.0.0.0
network.publish_host: ${HOST_IP}
cluster.name: "${CLUSTER_NAME}"

It's as if the elastic process isn't picking up the defaults file... If I use a fallback in the configuration (${HOST_IP:172.21.224.78}) it's able to start up...

Anything I'm doing wrong?

[ghost]

Oh sorry, I spoke too fast, I got it to work now! I just needed to EXPORT the variable in /etc/default/elasticsearch... It's working now with:

CLUSTER_NAME="RSD_ELK_PRD"
HOST_IP="172.21.224.78"
export CLUSTER_NAME
export HOST_IP

Thanks again, this is really useful to get a single image for multiple cluster members by dynamically assigning the publish_ip...

[spujadas]

Ah yes, thanks @yannouchou I'll update the documentation with the export bit.