Why doesn't `propagate-environment` include env vars set in hooks?

[evandam]

Hi folks,

Just wondering what the reason/limitation is behind why setting propagate-environment: true does not include environment variables that are set by hooks. The current behavior is clearly documented, but I'm unsure of why since it seems like it would be a very common and useful feature to have.

For example, I would expect the following to set GIT_SHA correctly:

/etc/buildkite-agent/pre-command:

export GIT_SHA="$(git rev-parse --short HEAD)"

pipeline.yml:

steps:
  - command: "echo $$GIT_SHA"
    env:
      FOO: bar
    plugins:
      - docker#v3.8.0:
          image: busybox
          propagate-environment: true

I need to pass environment: ["GIT_SHA"] explicitly in the plugin config for it to work. It would be helpful to include these environment variables, since a very common use of hooks is to expose environment variables to a step. It would also be very helpful in case a lot of env vars are set in hooks, and the steps don't need to keep the list in sync.

[pzeballos]

Hi @evandam,

For example, I would expect the following to set GIT_SHA correctly: /etc/buildkite-agent/pre-command:

I think the agent hook directory should be: /etc/buildkite-agent/hooks/pre-command

Can you confirm that the pre-command hook is definitely running? and if that is the case, could you send us some log output to confirm that everything is ok?

[evandam]

Hi @pzeballos,

Sorry - typo on my end. The hook is definitely running. Here's a better example that shows it working by explicitly passing the environment, and not working when just including propagate-environment:

steps:
  - command: "echo $$GIT_SHA"
    plugins:
      - docker#v3.8.0:
          image: busybox
          propagate-environment: true

  - command: "echo $$GIT_SHA"
    env:
      FOO: bar
    plugins:
      - docker#v3.8.0:
          image: busybox
          propagate-environment: true
          environment:
            - GIT_SHA

(pardon the horrible logs but copying straight from Buildkite output 😅)

First step logs (note there's no --env GIT_SHA):

$ docker run -it --rm --init --volume /var/lib/buildkite-agent/builds/buildkite-i-0c013ce9cc1a076a8-usw2-ci1-wonolo-shared-com-6/wonolo/ops-debug:/workdir --volume /var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-rds-snapshotter-git:/var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-rds-snapshotter-git:ro --workdir /workdir --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --env BUILDKITE_AGENT_ACCESS_TOKEN --volume /usr/bin/buildkite-agent:/usr/bin/buildkite-agent --env BUILDKITE_ORGANIZATION_SLUG --env BUILDKITE_AGENT_META_DATA_NAME --env BUILDKITE_AGENT_META_DATA_IPADDRESS --env BUILDKITE_AGENT_META_DATA_LOCAL_IPV4 --env BUILDKITE_BUILD_AUTHOR --env BUILDKITE_SCRIPT_PATH --env BUILDKITE_AGENT_META_DATA_CHEF_ENVIRONMENT --env BUILDKITE_BUILD_ID --env BUILDKITE_PIPELINE_PROVIDER --env BUILDKITE_AGENT_META_DATA_PLATFORM_FAMILY --env CI --env BUILDKITE_AGENT_META_DATA_LOCAL_HOSTNAME --env BUILDKITE_AGENT_META_DATA_REGION --env BUILDKITE_AGENT_META_DATA_PLATFORM_VERSION --env BUILDKITE_PROJECT_PROVIDER --env BUILDKITE_COMMAND --env BUILDKITE_BUILD_CREATOR_EMAIL --env BUILDKITE_STEP_ID --env BUILDKITE_PULL_REQUEST_REPO --env BUILDKITE_PLUGINS --env BUILDKITE_PIPELINE_NAME --env BUILDKITE_REPO_SSH_HOST --env BUILDKITE_BUILD_CREATOR_TEAMS --env BUILDKITE_PROJECT_SLUG --env BUILDKITE_REBUILT_FROM_BUILD_NUMBER --env BUILDKITE_RETRY_COUNT --env BUILDKITE_BUILD_AUTHOR_EMAIL --env BUILDKITE_REPO --env BUILDKITE_ARTIFACT_PATHS --env BUILDKITE_AGENT_META_DATA_FQDN --env BUILDKITE_COMMIT --env BUILDKITE_AGENT_ID --env BUILDKITE_LABEL --env BUILDKITE_TIMEOUT --env BUILDKITE_JOB_ID --env BUILDKITE_AGENT_META_DATA_AVAILABILITY_ZONE --env BUILDKITE_TAG --env BUILDKITE --env BUILDKITE_MESSAGE --env BUILDKITE_BUILD_NUMBER --env BUILDKITE_TRIGGERED_FROM_BUILD_NUMBER --env BUILDKITE_AGENT_META_DATA_PLATFORM --env BUILDKITE_PULL_REQUEST_BASE_BRANCH --env BUILDKITE_BUILD_CREATOR --env BUILDKITE_PIPELINE_SLUG --env BUILDKITE_PULL_REQUEST --env BUILDKITE_SOURCE --env BUILDKITE_AGENT_META_DATA_OS_VERSION --env BUILDKITE_TRIGGERED_FROM_BUILD_ID --env BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG --env BUILDKITE_AGENT_META_DATA_QUEUE --env BUILDKITE_AGENT_META_DATA_INSTANCE_ID --env BUILDKITE_AGENT_META_DATA_ACCOUNT_ID --env BUILDKITE_REBUILT_FROM_BUILD_ID --env BUILDKITE_STEP_KEY --env BUILDKITE_BRANCH --env BUILDKITE_AGENT_NAME --env BUILDKITE_BUILD_URL --env BUILDKITE_PIPELINE_ID --env BUILDKITE_AGENT_META_DATA_INSTANCE_TYPE --env BUILDKITE_PIPELINE_DEFAULT_BRANCH --label com.buildkite.job-id=b46e9e01-218e-4a3e-9896-6ab7f1af30a3 busybox /bin/sh -e -c echo\ \$GIT_SHA
--

 

Second step logs:

$ docker run -it --rm --init --volume /var/lib/buildkite-agent/builds/buildkite-i-0073d9ef9b31c22ac-usw2-ci1-wonolo-shared-com-13/wonolo/ops-debug:/workdir --volume /var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-rds-snapshotter-git:/var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-rds-snapshotter-git:ro --workdir /workdir --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --env BUILDKITE_AGENT_ACCESS_TOKEN --volume /usr/bin/buildkite-agent:/usr/bin/buildkite-agent --env GIT_SHA --env BUILDKITE_PULL_REQUEST_REPO --env BUILDKITE_AGENT_META_DATA_LOCAL_HOSTNAME --env BUILDKITE_REBUILT_FROM_BUILD_ID --env BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG --env BUILDKITE_AGENT_META_DATA_INSTANCE_ID --env BUILDKITE_PULL_REQUEST_BASE_BRANCH --env BUILDKITE_PROJECT_SLUG --env BUILDKITE_BUILD_CREATOR --env BUILDKITE_PIPELINE_NAME --env BUILDKITE_AGENT_META_DATA_PLATFORM --env BUILDKITE_BUILD_ID --env BUILDKITE_BUILD_URL --env BUILDKITE_COMMAND --env BUILDKITE_AGENT_META_DATA_AVAILABILITY_ZONE --env BUILDKITE_AGENT_META_DATA_ACCOUNT_ID --env BUILDKITE_PROJECT_PROVIDER --env BUILDKITE_ORGANIZATION_SLUG --env BUILDKITE_AGENT_NAME --env BUILDKITE_STEP_ID --env FOO --env BUILDKITE_AGENT_ID --env BUILDKITE_PIPELINE_SLUG --env BUILDKITE_RETRY_COUNT --env BUILDKITE_REBUILT_FROM_BUILD_NUMBER --env BUILDKITE_SCRIPT_PATH --env BUILDKITE_JOB_ID --env BUILDKITE_AGENT_META_DATA_OS_VERSION --env BUILDKITE --env BUILDKITE_AGENT_META_DATA_NAME --env CI --env BUILDKITE_TRIGGERED_FROM_BUILD_ID --env BUILDKITE_AGENT_META_DATA_LOCAL_IPV4 --env BUILDKITE_TAG --env BUILDKITE_COMMIT --env BUILDKITE_AGENT_META_DATA_PLATFORM_FAMILY --env BUILDKITE_BRANCH --env BUILDKITE_AGENT_META_DATA_CHEF_ENVIRONMENT --env BUILDKITE_TRIGGERED_FROM_BUILD_NUMBER --env BUILDKITE_PIPELINE_PROVIDER --env BUILDKITE_BUILD_CREATOR_EMAIL --env BUILDKITE_PLUGINS --env BUILDKITE_BUILD_CREATOR_TEAMS --env BUILDKITE_AGENT_META_DATA_PLATFORM_VERSION --env BUILDKITE_ARTIFACT_PATHS --env BUILDKITE_AGENT_META_DATA_INSTANCE_TYPE --env BUILDKITE_LABEL --env BUILDKITE_PIPELINE_ID --env BUILDKITE_REPO_SSH_HOST --env BUILDKITE_BUILD_AUTHOR --env BUILDKITE_TIMEOUT --env BUILDKITE_BUILD_AUTHOR_EMAIL --env BUILDKITE_REPO --env BUILDKITE_STEP_KEY --env BUILDKITE_PULL_REQUEST --env BUILDKITE_PIPELINE_DEFAULT_BRANCH --env BUILDKITE_AGENT_META_DATA_FQDN --env BUILDKITE_SOURCE --env BUILDKITE_AGENT_META_DATA_REGION --env BUILDKITE_AGENT_META_DATA_IPADDRESS --env BUILDKITE_AGENT_META_DATA_QUEUE --env BUILDKITE_MESSAGE --env BUILDKITE_BUILD_NUMBER --label com.buildkite.job-id=bed44259-537e-4c20-967e-292b4005a89c busybox /bin/sh -e -c echo\ \$GIT_SHA
--

8c8717c

Thanks!

[pzeballos]

Hi @evandam,

Sorry for the delay; we were discussing your issue with the rest of the team. What you point out makes sense, and it would be logical to include environment variables exported from preceding hooks. It only propagates the standard list of job environment variables, and ones defined in the pipeline.yml, because it uses $BUILDKITE_ENV_FILE:

https://github.com/buildkite-plugins/docker-buildkite-plugin/blob/master/hooks/command#L262-L264

But this sounds like an excellent suggestion, and it would be a great feature enhancement if you like to submit a PR, we would love that!

Cheers!

[evandam]

Hi @pzeballos no worries! I don't think I'm familiar enough with the Buildkite internals to be the one to open a PR 😅

It feels like blindly propagating every env var may not be the best idea since it would stomp on things like $PATH, $HOME, etc. I'm not sure if there's a way to know if an env var was exported by a hook or not?

[bluemalkin]

This is a real limitation, can we please get this implemented ?

My use case is I need to make some secrets available to builds in docker plugins and currently I cannot find a way to do this.

[toote]

As far as I understand, there is no way for this plugin to know what variables have been set by other plugins throughout the step's execution history so it would not be possible to create that list to be added in the propagate-environment option or a new one implemented to do just that.

That is why the Readme specifically states that it can not even share environment variables created by the S3 secrets plugin included in the AWS elastic stack. Which means you would have to resort to specifically adding variables to share one by one... for which the environment parameter exists