Closed memotoro closed 6 months ago
@memotoro sorry for the delay in getting back to you!
I don't think that it would be easy to add to the plugin a way to specify what environment variables should be shared with each container. Luckily, docker compose has a way to do that. Have you tried adding environment
stanzas to the services themselves? That way, if the variables are available in the context they will be used by docker compose and would not need any modification of the plugin. It would also serve as documentation on your services and dependencies as to what each need :)
Hi @toote , thanks for the reply. I had the other docker-compose
files with environment
section per service. I also have the env
section in buildkite
with the definition of the ENV and values. However, that doesn't work either. It looks like with the plugin is kicking off the docker compose -f ..... up -d --scale tests=0 tests
doesn't resolve ENVs defined in the env
from buildkite
, it only works if the envs
in the environment
section of the docker-compose
file are hardcoded, but that defeat the purpose on some ENVs. I tried to propagate with buildkite
flags but that doesn't work either. I'm using docker
directly on bash for the time being, but wondering why the plugin is not passing the envs when doing the scale to 0.
@memotoro sorry for the time it took me to get back to you but unfortunately I have not been able to duplicate your issue.
I used the following composefile:
services:
db:
image: bash:4.2
command: bash -c 'env; sleep 60'
environment:
- TEST_VARIABLE2
- TEST_VARIABLE3
app:
image: ruby
depends_on:
- db
environment:
- TEST_VARIABLE
And the following step in the pipeline:
steps:
- label: test
command: env
plugins:
- docker-compose#v4.14.0:
run: app
config: compose.yaml
shell: false
upload-container-logs: always
If the agent (or pipeline) I defined the variables TEST_VARIABLE
and TEST_VARIABLE3
and the output of the step was:
$ docker-compose -f compose.yaml -p buildkite018d80c7a0374967a1aaf7f14effa393 up -d --scale app=0 app
Creating network "buildkite018d80c7a0374967a1aaf7f14effa393_default" with the default driver
Creating buildkite018d80c7a0374967a1aaf7f14effa393_db_1 ... done
:docker: Running 'env' in service app
$ docker-compose -f compose.yaml -p buildkite018d80c7a0374967a1aaf7f14effa393 run --name buildkite018d80c7a0374967a1aaf7f14effa393_app_build_31 [SNIP] --rm app env
Creating buildkite018d80c7a0374967a1aaf7f14effa393_app_run ... done
PATH=/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=b6ee35d65eb2
TERM=xterm
TEST_VARIABLE=testvalue
LANG=C.UTF-8
RUBY_MAJOR=3.2
RUBY_VERSION=3.2.2
RUBY_DOWNLOAD_SHA256=4b352d0f7ec384e332e3e44cdbfdcd5ff2d594af3c8296b5636c710975149e23
GEM_HOME=/usr/local/bundle
BUNDLE_SILENCE_ROOT_WARNING=1
BUNDLE_APP_CONFIG=/usr/local/bundle
HOME=/root
And the step had an artifact with the logs for the db
dependency:
2024-02-06T23:36:31.561956162Z HOSTNAME=8d57b4ee6d15
2024-02-06T23:36:31.561987242Z _BASH_BASELINE=4.2.53
2024-02-06T23:36:31.561991873Z TEST_VARIABLE3=testvalue3
2024-02-06T23:36:31.561995257Z _BASH_BASELINE_PATCH=53
2024-02-06T23:36:31.561998408Z PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-02-06T23:36:31.562002152Z PWD=/
2024-02-06T23:36:31.562005214Z _BASH_VERSION=4.2.53
2024-02-06T23:36:31.562008198Z HOME=/root
2024-02-06T23:36:31.562011533Z SHLVL=1
2024-02-06T23:36:31.562014557Z _BASH_LATEST_PATCH=53
2024-02-06T23:36:31.562017612Z _=/usr/bin/env
As you can see TEST_VARIABLE
is available and correctly defined in the main container being run and TEST_VARIABLE3
in the dependency container (but not TEST_VARIABLE2
because it was not defined) despite the plugin being run with no modifications. So I am not sure why a similar setup is not working for you :(
Hello
Just checking the behaviour of the plugin
v4.14.0
and see if that is something that could be changed.I have 2 docker compose files.
The
docker-compose-dependencies.yaml
contains 2 containers that are required for the test to run. Let's call it containerapi
and containerdb
. These two containers will require specific environment variables. These containers are usinghealthchecks
to make sure containers are correctly working before accessing them.The
docker-compose-test.yaml
contains only the container with the test framework (let's call ittests
) to test the containerapi
. Thistests
container also requires specific environment variables.It looks like the plugin kicks off the dependencies on the file
docker-compose-dependencies.yaml
first (containersapi
anddb
), if the flagrun_dependencies
is set totrue
, then those containers are started with a command similar to the following:Later, the plugin runs the
testing
container in the filedocker-compose-test.yaml
specified in therun
property. It looks like the environment variables are only passed to the container specified in therun
command, similar to the following:I wonder if the environment variables cannot be propagated to the command that triggers the dependencies also. When using multiple configuration files, other containers need also configuration fed via environment variables. However, if those environment variables are not passed to the command shown above, then those container will not start correctly as some environment variables are missing, hence
healthchecks
fail and then the whole testing suite will fail. It would be nice if the dependencies are also fed with a command similar to the following:I have tried to use the different flags to propagate environment variables, but it doesn't propagate the environment variables that I have set in the
env
attribute of the plugin or in theenv
attribute of the buildkite step command.I have also set the property
run_dependencies
as false. In this case, the plugin starts all containers at the same time but only feed the environments to the container specified in therun
property.Is this the correct or expected behaviour ?
Thanks for any advice.
Tagging @toote @pzeballos as most recent contributors.