Interpolate environment variables in docker-compose.yml

[aanand]

(I'm creating a fresh issue for this as the old one has accumulated rather a lot of baggage.)

It should be possible to pass in environment variables to the value of any* configuration entry in docker-compose.yml. A lot of people want to do it, it's good for portability and I'm satisfied it's not going to cause chaos.

I have some reckons.

Required variables and optional defaults

It's useful to be able to specify that a variable that must be present in the environment, i.e. that Compose will refuse to run if it isn't. However, this will be a pain when you've got lots of them, so it should either be something you explicitly enable, or it should be possible to specify a default value.

The MVP implementation does not need to have either feature, but there should be a clear path to implementing both in a backwards-compatible way.

Syntax

There's a strong case for implementing an established standard, as long as it's not heavyweight - our requirements for functionality are minimal.

• POSIX parameter expansion is OK. It has a few too many features, but we could implement a subset of them:

  • ${VARIABLE} - outputs empty string if VARIABLE is unset
  • ${VARIABLE-default} - outputs default if VARIABLE is unset
  • ${VARIABLE?} - errors out if VARIABLE is unset

  https://github.com/docker/compose/pull/845 implemented a Bash-style ${VARIABLE:default} syntax, which is similar to POSIX parameter expansion but slightly different.

• Python's format syntax would be trivial to implement, but is tied to the Python language, and has rather too many features.

  Implementation

Python's os.path.expandvars function implements the most basic case of POSIX parameter expansion:

>>> from os.path import expandvars
>>> expandvars('${HOME}')
'/Users/aanand'

However, it's got at least 2 problems:

1. An unset variable doesn't expand to an empty string - instead, it results in no expansion:

   >>> expandvars('${UNSET}')
   '${UNSET}'

2. Malformed syntax doesn't error out - instead, it also results in no expansion:

   >>> expandvars('${HOME')
   '${HOME'

So far, https://github.com/docker/compose/pull/845 is the closest we've got, but I'm fundamentally wary of an implementation that relies on regular expressions. Templating is a non-trivial job, and people are going to put all kinds of broken stuff in, so we need something that's robust, strict and errors out with helpful messages. Two important requirements:

• If someone puts in something malformed, Compose will not run.
• It is possible to escape any of the special characters used in the template syntax.

There may well be good Python implementations of Bash-like variable interpolation out there already - if not, creating something standalone would be far preferable to bloating the Compose codebase.

*Actually, are there any configuration keys for which we shouldn't allow interpolation?

[kojiromike]

How far do you want to go with these established UNIX standards? (FWIW, it's not a defacto standard, it's an actual standard.)

As someone who occasionally accidentally tries to use POSIX parameter expansions in Dockerfiles, if they were at all supported in docker-compose.yml it would make me a happy camper.

[aanand]

@kojiromike Hmm, so POSIX parameter expansion is actually what I was going for, but reading over the docs it looks like I misremembered the syntax/semantics.

Edit: I've updated my thoughts on syntax in the description.

[Vad1mo]

I have been following the old thread and we urgently wanted to have this feature. Finally the pain was too big and we created a yaml preprocessor bahs script to substitute variables in POSIX style. it worked fine but eventually we stopped using it, because it had one issue. You have to run the preprocessor first and set all the parameters before you get the final solution. Now we are using the docker yaml extends feature. Because it allows us to checkin the actual configuration and just execute it on the target. We know better what is going to happen.

Even though I was a supporter of docker-compose passing variables, I am now not so sure.

As an ideal solution I would rather see docker extends done right. In a sense that would be a solution that fits both. So what is broken in docker extends? It is basically the fact that you have to write all entries in the inherited file. It is not a merge where you enter only what you want to override.

Look at out actual example and how verbose it is. There are only two lines that matter.

#Common 
elasticsearch:
  image: zinvoice/elasticsearch
  hostname: elasticsearch
  restart: always
  dns: 172.17.42.1
  ports:
    - "9200:9200"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - /data/elasticsearch:/opt/elasticsearch/data/elasticsearch

logstash:
  image: zinvoice/logstash
  hostname: logstash
  dns: 172.17.42.1
  restart: always
  ports:
    - "5000:5000"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

kibana:
  image: zinvoice/kibana
  hostname: kibana
  dns: 172.17.42.1
  restart: always
  ports:
    - "5601:5601"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

logspout:
  image: zinvoice/logspout
  hostname: logspout
  command: logstash://logstash.docker:5000
  restart: always
  dns: 172.17.42.1
  ports:
    - "8003:8000"
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock

doorman:
  image: zinvoice/doorman
  hostname: doorman
  restart:  always
  dns: 172.17.42.1
  ports:
    - "8085:8085"

# inherited
elasticsearch:
  extends:
    file: ../common.yml
    service: elasticsearch

logstash:
  extends:
    file: ../common.yml
    service: logstash

kibana:
  extends:
    file: ../common.yml
    service: kibana

logspout:
  extends:
    file: ../common.yml
    service: logspout

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx
  links:
    - nginxtrusted
  extends:
    file: ../common.yml
    service: doorman

So my recommendation fix docker extends an make it less verbose. You don't even have to write that much code as YAML provides all the functionality you need. If you stick with standard YAML the file could be analyzed or created by other tools and UIs.

Take a look at YAML "node anchors" and YAML "file merge" it might be the perfect solution.

FYI: this discussion continuous now on #1380

[aanand]

@Vad1mo I agree that extends falls short in your case. There are a lot of things we can do to improve that experience - could you open a separate issue for it, so we don't get sidetracked here?

[Vad1mo]

Of course! I just wanted to highlight that this could be an easy and elegant alternative. If compose extends gets you half the way too variable passing, then an improved compose-extends will make variable passing obsolete. Having less concepts to understand make easier for the user.

[mattes]

My use case is to allow $PWD in volumes, so every developer in the team can clone a repo to wherever and paths still get mounted correctly.

elasticsearch:
  image: zinvoice/elasticsearch
  volumes:
    - $PWD:/app

[dnephin]

@mattes I believe that is already supported, I think .:/app is supported as well

[kojiromike]

@aanand As a PoC I did a dirty hackup of POSIX PE in Python. For the Saturdays.

[aanand]

@kojiromike Looks great. Let me know if you plan to continue on it.

[kojiromike]

@aanand I intend to, but it definitely has a few bugs right now (and I think it may have been a bad idea to use shlex). Bug reports and PRs are welcome, of course.

[nafg]

@dnephin how about $HOME / ~?

[dnephin]

@nafg Both of those are supported for the host path of a volume

[nafg]

@dnephin interesting, b/c somehow I ended up with a directory named '$HOME'...

[andrerom]

@aanand Like the "${VARIABLE:default}" proposal, with global_extends (or "import") this would become rather powerful.

Q: Would this allow to specify port number that is exposed to host? like - "${WEB_PORT:80}:80"? Use case is to be able to easily spin up several instances of an app on same machine/cluster, typically listening to different ports or assigned to different local domain names.

[aanand]

Yes, you'd be able to do that.

[alexanderilyin]

I'd like to use vars in volumes together with docker-compose scale my_app=3. I have this docker-compose.yml

server:
  image: alexanderilyin/docker-teamcity-server
  ports:
   - "8111:8111"
  volumes:
    - .TeamCity:/root/.BuildServer
  links:
   - mysql
mysql:
  image: alexanderilyin/docker-mysql
  volumes:
    - .MySQL:/var/lib/mysql
  environment:
    MYSQL_DATABASE: teamcity
    MYSQL_USER: teamcity
    MYSQL_PASSWORD: teamcity
    MYSQL_ALLOW_EMPTY_PASSWORD: yes
agent:
  image: alexanderilyin/docker-teamcity-agent
  links:
   - server

And I want to be able use scale for agents and use dynamical volumes for them to keep data between launches, for example:

agent:
  image: alexanderilyin/docker-teamcity-agent
  volumes:
    - .agent_{$AGENT_INSTANCE_ID}:/opt/buildAgent
  links:
   - server

[Maxim-Filimonov]

I hope it would be possible to interpolate variables as part of image name too We are using https://github.com/openshift/source-to-image to build local container on CI for every branch and then run tests on it using docker-compose. Running tests with dynamic image is quite complicated with docker-compose and requires manual template rendering.. :-1:

[andrerom]

But you can set COMPOSE_PROJECT_NAME to control prefix per run to be able to do that already right? If so, no need to have complex logic and unreadable yml files around names.

[Maxim-Filimonov]

@andrerom don't follow. According to docs that controls the following Sets the project name, which is prepended to the name of every container started by Compose while we are trying to set an image property instead:

web:
  image: <I_AM_DYNAMIC>

[andrerom]

ah, my mistake.

Thought you meant

<I_AM_DYNAMIC>:
  image: nginx

Dynamic image (and build) reference would indeed make a lot of sense. For instance switching between debug and non debug base containers for your programming language for instance would be a good use case for this.

[andrerom]

Additional use case (which might be what @Maxim-Filimonov has in mind): Being able to override which tag to use of an image, so you can use :latest by default, but change to easily test something else without changing yml file (needed for CI use cases basically).

[Maxim-Filimonov]

@andrerom that is exactly our use case :+1:

[ghost]

Will this also work for things like??

web:
  environment:
    - FOO=${whoami}

[thaJeztah]

@k0377 I don't think they will, because that's really something that's handled by the shell, but you could add the result in an environment variable and use that.

In this particular case, the $USER environment variable will probably give you the same.

[askoretskiy]

@aanand Why not use any of existing template engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task (and regexps are not that cool) so that we should use already existing one, that proven to be solid.

[askoretskiy]

Alternatively we might use YAML ancors and references https://gist.github.com/bowsersenior/979804

But then we are limited on variables usage (inject variable name into middle of content).

[DoubleMalt]

+1 for Jinja2: it would certainly fit the mold and ansible uses it for exactly that use case (templating in yml files)

On Tue, May 26, 2015 at 1:25 PM, tonnzor notifications@github.com wrote:

@aanand https://github.com/aanand Why not use any of existing template engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task (and regexps are not that cool) so that we should use already existing one, that proven to be solid.

— Reply to this email directly or view it on GitHub https://github.com/docker/compose/issues/1377#issuecomment-105493447.

[aanand]

Jinja2 does a lot more than we need:

• conditionals
• looping
• extension/inheritance
• comments
• filters

We're not adding any of that stuff to Compose. If Jinja2 can be configured to just interpolate variables, then it might be a candidate.

[DoubleMalt]

Actually looping might be interesting.

Assume you have a list of customers that you want to start containers for where you put some customer specific variables into the environment.

Extension/Inheritance might be interesting to enhance the current rudimentary extension mechanism.

Filters can be great to do something with existing variables.

On Tue, May 26, 2015 at 1:56 PM, Aanand Prasad notifications@github.com wrote:

Jinja2 does a lot more than we need:

• conditionals
• looping
• extension/inheritance
• comments
• filters

We're not adding any of that stuff to Compose. If Jinja2 can be configured to just interpolate variables, then it might be a candidate.

— Reply to this email directly or view it on GitHub https://github.com/docker/compose/issues/1377#issuecomment-105498909.

[aanand]

They might be interesting features, but they come with far more complexity than I'm comfortable introducing to both Compose and the file format, and we'd be tying both to a specific templating language with (as far as I know) a single implementation and no spec. It's simply not feasible.

[askoretskiy]

@aanand Some notes here:

1. Jinja2 is solid and it takes minutes to do pre-processing of YAML with it:

from jinja2 import Template template = Template('Hello {{ name }}!') template.render(name="Aanand") Hello Aanand!

If you want more security -- you could use immutable sandbox:

from jinja2.sandbox import ImmutableSandboxedEnvironment env = ImmutableSandboxedEnvironment() template = env.from_string('Hello {{ name }}!') template.render(name="Aanand") Hello Aanand!

In our case it would be:

import os from jinja2.sandbox import ImmutableSandboxedEnvironment env = ImmutableSandboxedEnvironment() template = env.from_string('Hello {{ name }}!') template.render(**os.environ)

1. Don't we want filters? With filter you could define default value easily ( {{ value|default("default") }} )
2. Do we really need to care about users that use extended Jinja features to screw YAML file? In the same manner user could produce invalid YAML file manually. I think we should keep it simple -- try to process given Jinja template and return error if there was an error or produced YAML is invalid (the same as you do now).
3. If you don't see Jinja2 as solution -- it would be great at least to use {{ variable }} as syntax.
4. Django uses regexp to parse and generate template. It is production-grade for a long time and lives fine with it.

import os import re template = "Hello {{ name }}!" re.sub("{{\s([a-zA-Z0-9]+?)\s_}}", lambda m: os.environ.get(m.group(1), ''), template)

In any case -- we need to have this feature rolling, whatever solution we take.

[thaJeztah]

I'm +1 on using a generic templating solution if templates are considered. E.g. http://mustache.github.io, which is available in many languages. This is just an example, other templating engines can be considered equally

[DoubleMalt]

@aanand I totally understand your point. I also like the simplicity and succinctness of the compose dsl.

Maybe this should be done as an external project, say meta-composer. It takes a compose.tpl.yml and a variables.yml, creates a docker-compose.yml and off we go. As @tonnzor showed t could be done with a tiny piece of python code.

This would deliver powerful templating to those who need it without introducing complexity for simple tasks.

On Tue, May 26, 2015 at 4:52 PM, Sebastiaan van Stijn < notifications@github.com> wrote:

I'm +1 on using a generic templating solution if templates are considered. E.g. http://mustache.github.io, which is available in many languages. This is just an example, other templating engines can be considered equally

— Reply to this email directly or view it on GitHub https://github.com/docker/compose/issues/1377#issuecomment-105551631.

[kojiromike]

Hmm… So is the proposal now to use a templating language inside compose.yml (which is a descriptive language for composing Docker containers), for things like command and entrypoint, which already accept both exec and sh -c style values? This could be confusing, since after template rendering the resultant shell command would still presumably be interpreted, so if a variable happened to expand to * it would further be glob expanded. Escaping sequences in one language or another becomes tricky when you have this many layers of fall-through interpretation.

[thaJeztah]

@kojiromike I'm not sure a templating engine is wanted, but if it's to be used! better use something well known. The basic question is; should docker-compose write the substitution from-scratch, or use something existing.

[nafg]

On Tue, May 26, 2015, 11:02 AM Christoph Witzany notifications@github.com wrote:

@aanand I totally understand your point. I also like the simplicity and succinctness of the compose dsl.

Maybe this should be done as an external project, say meta-composer. It takes a compose.tpl.yml and a variables.yml, creates a docker-compose.yml and off we go.

You can do that today without any new project. I'm sure jinja can be invoked from the command line somehow. Personally I just use the envsubst command.

What would be really helpful is if compose could read the file from stdin. That would eliminate the need for an intermediate file.

As @tonnzor showed t could be done with a tiny piece of python code.

This would deliver powerful templating to those who need it without introducing complexity for simple tasks.

On Tue, May 26, 2015 at 4:52 PM, Sebastiaan van Stijn < notifications@github.com> wrote:

I'm +1 on using a generic templating solution if templates are considered. E.g. http://mustache.github.io, which is available in many languages. This is just an example, other templating engines can be considered equally

Reply to this email directly or view it on GitHub https://github.com/docker/compose/issues/1377#issuecomment-105551631.

Reply to this email directly or view it on GitHub https://github.com/docker/compose/issues/1377#issuecomment-105554730.<img src=" https://ci6.googleusercontent.com/proxy/iSBXyl7D8PwFM4p1mGPHCR7bQctunieGbhyGkvo0QIMIjmAYE3I0Mt96yl1fGrqcuOzxV4APP8ZRIw-5_qd6nzps9Mpr6jTAydCC4xs8JDgqm93aIbWvN1eMlxykrz7iwYooyAQdqL4RFJokeEbnBkZm5mhgKg=s0-d-e1-ft#https://github.com/notifications/beacon/AAGAUO8xqz29B2SUoG7QFPUy848_JJW9ks5oNIJlgaJpZM4EMysO.gif ">

[kevana]

+1 for reading the file from stdin. I have no problem with using an external template solution but not having intermediate files around would be nice.

[dnephin]

That sounds like a great first step, and a common feature of many cli tools. Let's do that

[Maxim-Filimonov]

:+1:

[kojiromike]

So for example

envsubst compose.tmpl.yml | docker-compose -f - up -d

wfm. :+1:

[thaJeztah]

Just noticed that docker/distribution handles overriding values in the yml file via environment variables, but using a different approach https://github.com/docker/distribution/blob/master/docs/configuration.md#override-configuration-options

^^ @aanand

[Maxim-Filimonov]

@thaJeztah that would work for us too. We can use environment variables to override commands then

DOCKER_COMPOSE_IMAGE_NAME='my_image:is_dynamic'

[aanand]

Interesting approach, but I'm not a fan - verbose environment variable names, lots of duplication if you want to use one value in multiple places, everything's implicit, no interpolation within strings.

[thaJeztah]

@aanand not really sold on that approach either, but wanted to point to it because its another project within the "Docker" organization.

[olalonde]

Just stumbled on https://github.com/kelseyhightower/confd which might be of interest. It uses http://golang.org/pkg/text/template/#pkg-overview

[thaJeztah]

@olalonde unfortunately, docker-compose is written in Python, so Go-templates won't work.

[andrerom]

@aanand I'm +20 on your original proposal, with the tweak that even images and especially tags should be possible to inject. Just go for it, would save all of us a lot of wrappers and unneeded config handling ;)

[DoubleMalt]

I wrote a little python package that helps me with that. The plan is to tunnel all commands through to docker compose so you can use it equivalently. Check it out at https://github.com/webcrofting/meta-compose/

[hadim]

meta-compose looks really nice. It should be integrated to docker-compose !

[smari]

Big +1 here -- I'm not excited about template preprocessing, but pulling environment variables one way or another would be great. POSIX expansion is probably cleaner than Jinja2, but either way is fine.