Inconsistency exposing shell environment variables between macOS and Ubuntu

[vmrob]

Per the docker-compose docs, docker-compose run -e FOO service should expose the environment variable FOO from the host to the service. This works on macOS:

$ docker --version
Docker version 1.13.1, build 092cba3
$ docker-compose --version
docker-compose version 1.11.1, build 7c5d5e4
$ cat docker-compose.yml
version: '2'
services:
    foobar:
        image: ubuntu:16.04
$ export FOO=asdf
$ docker-compose run --rm -e FOO foobar
root@e3d159a32d22:/# echo $FOO
asdf

The latest on Ubuntu does not produce the same behavior:

$ docker --version
Docker version 1.13.1, build 092cba3
$ docker-compose --version
docker-compose version 1.11.1, build 7c5d5e4
$ cat docker-compose.yml
version: '2'
services:
    foobar:
        image: ubuntu:16.04
$ export FOO=asdf
$ docker-compose run --rm -e FOO foobar
Creating network "temp_default" with the default driver
root@8ddf7da98a76:/# echo $FOO

Both work if I set the variable manually with -e FOO=asdf.

I'm running Ubuntu 16.04 from Amazon (ami-40d28157)

I believe I've also seen issues using .env, but I'm not sure enough to put together a minimal repro.

[shin-]

Thanks for the report, I'll look into it.

[shin-]

Hi @vmrob ,

I can't reproduce the issue you describe on my machine running Ubuntu 16.04 (Desktop - not that it should matter). Can you think of anything special about your AMI setup that could cause this issue? How did you install docker-compose? Does the env command on the host show the FOO variable after you export it? What do you get if you run python -c 'import os; print os.environ.get("FOO")'?

[vmrob]

It should. I actually built the AMI with packer, so I could provide a fully-reproducible build, but let's try that first.

[vmrob]

Ah, great. Here's a Vagrantfile that should demonstrate the problem:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/xenial64"

  config.vm.provision "shell", inline: <<-SHELL
    set -x

    apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
    echo 'deb https://apt.dockerproject.org/repo ubuntu-xenial main' > /etc/apt/sources.list.d/docker.list

    apt-get update
    apt-get install -y docker-engine python

    curl -L https://github.com/docker/compose/releases/download/1.11.1/run.sh > /tmp/docker-compose
    mv /tmp/docker-compose /usr/local/bin/docker-compose
    chmod +x /usr/local/bin/docker-compose

    docker --version
    docker-compose --version

    cat << EOF > docker-compose.yml
version: '2'
services:
  foobar:
      image: ubuntu:16.04
EOF
  SHELL

  config.vm.provision "shell", inline: <<-SHELL
    set -ex
    export FOO=bar

    # each of the following should emit 'bar', set -e ensures that if it
    # doesn't, the script fails
    echo $FOO | grep bar
    python -c 'import os; print os.environ.get("FOO")' | grep bar
    docker-compose run --rm -e FOO=$FOO foobar bash -c 'echo $FOO' | grep bar
    docker-compose run --rm -e FOO foobar bash -c 'echo $FOO' | grep bar
  SHELL
end

Relevant output on my end:

==> default: Running provisioner: shell...
    default: Running: inline script
==> default: mesg: ttyname failed: Inappropriate ioctl for device
==> default: ++ export FOO=bar
==> default: ++ FOO=bar
==> default: ++ echo bar
==> default: ++ grep bar
==> default: bar
==> default: ++ python -c 'import os; print os.environ.get("FOO")'
==> default: ++ grep bar
==> default: bar
==> default: ++ docker-compose run --rm -e FOO=bar foobar bash -c 'echo $FOO'
==> default: ++ grep bar
==> default: bar
==> default: ++ docker-compose run --rm -e FOO foobar bash -c 'echo $FOO'
==> default: ++ grep bar
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

Note that the last grep failed as the command didn't produce the string bar.

EDIT: that vagrantfile might require running twice.. fixing it.. EDIT: fixed

[vmrob]

Any luck reproducing it with the provided Vagrantfile?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mikavilpas]

I think I also was affected by this. I am able to work around it by not using the run.sh wrapper script (the one that runs docker-compose in a container), and instead running docker-compose from the system's (ubuntu/linux mint) package. I can reproduce this with:

# Dockerfile
FROM alpine

# docker-compose.yml
version: '3'

services:
  test:
    build: .
    environment:
      FOO: $FOO
    command: echo $FOO

shell commands:

# Environment variables cannot be passed with the provided run.sh wrapper script
# https://docs.docker.com/compose/install/#install-as-a-container
$ FOO=bar ./run.sh up
WARNING: The FOO variable is not set. Defaulting to a blank string.
Recreating tmpqzs8dexfli_test_1 ... done
Attaching to tmpqzs8dexfli_test_1
test_1  |
tmpqzs8dexfli_test_1 exited with code 0

# my (ubuntu) docker-compose works
$ FOO=bar docker-compose up
Starting tmpqzs8dexfli_test_1 ...
Starting tmpqzs8dexfli_test_1 ... done
Attaching to tmpqzs8dexfli_test_1
test_1  | bar
tmpqzs8dexfli_test_1 exited with code 0

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.

[mikavilpas]

Should I open a new issue? Looks like there are a lot of issues and I'm not sure if this will get noticed because it's been open for such a long time.

[ndeloof]

Hi. Please don't open another issue, as long as this one is opened it should be considered.

[ndeloof]

https://github.com/docker/compose/blob/master/script/run/run.sh#L63 do exec docker run to run compose within a container but does not propagate local environment variables. Not sure how to get a clean fix for this, as we probably don't want to just propagate anything from local environment (typically: PATH or HOME would make no sense.

[mikavilpas]

I have a hacky idea and would like to hear your comments. Here is the gist of it:

Instead of propagating everything, parse all given docker-compose files and see which variables they require. Then pass all those from the current environment.

I'm thinking some hacky regex for the $foo and ${bar} cases would work, but if something better than a regex is available then it should be used.

If you think this is okay, I might be able to work on this on company time next week or so. We have an issue related to this and would benefit from fixing it.

[ndeloof]

run.sh should really just be a wrapper to bootsrap compose inside a container. Adding some logic here to parse the compose file and identify variables would just create a two heads beast, any change to the compose file format hacing to be implemented twice (we actually already have to maintain both docker-compose and compose support in docker cli, please don't add a new bash version or if :P)

[mikavilpas]

Okay, I see.

I'm thinking of other ways to fix it. How about this idea:

The run.sh wrapper captures the environment as a file (env > top-level.env) and shares that file to the container running docker-compose. Then docker-compose could read and apply only the variables it needs.

I haven't used .env overrides with docker-compose yet, so I might not see some possible error scenarios here. What do you think, could something like that work?

[ulyssessouza]

@sp3ctum You can already pass the variables you need through DOCKER_RUN_OPTIONS environment variable, like in:

$ export FOO="myfoovalue"
$ export BAR="mybarvalue"
$ DOCKER_RUN_OPTIONS="-e FOO -e BAR" ./script/run/run.sh up

@vmrob As for your original issue I could not reproduce, I'm closing it