Update DD_API_KEY during the docker runtime

[aaband]

Describe what happened: I want to put dd agent as a sidecar container inside the circle ci job. Example:

executors:
  with-chrome:
    working_directory: /home/cypress
    docker:
      - image: us.gcr.io/cypress/cypress:latest
        environment:
          DD_ENV: ci
          DD_SERVICE: app-service
      - image: gcr.io/datadoghq/agent:latest
        environment:
          DD_INSIDE_CI: "true"
          DD_API_KEY: apikeyinplaintext
          DD_HOSTNAME: "none"

Describe what you expected: I expect the container to start and reread DD_API_KEY env value as it changes. In circleci env variables are populated as a step only after the docker container start, but dd agent container can't start without env var provided, container exits with code 1. So I cant use vars from circleci environment variables. The only way to go is to put api_key var in plain text in the github repository.

I use the job defined in the 3rd party orb, I cant incorporate agent installation there as a step. Steps to reproduce the issue:

docker run -it --rm gcr.io/datadoghq/agent:7   
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-check-apikey.sh: executing... 

==================================================================================
You must set an DD_API_KEY environment variable to run the Datadog Agent container
==================================================================================

[cont-init.d] 01-check-apikey.sh: exited 1.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.

[fermayo]

Hello @aaband, thanks for reaching out.

As you pointed out, CircleCI does not inject project environment variables into secondary containers (feature request), so the agent cannot be run as one.

The current workaround we use in our Datadog Agent Orb is to install the agent in the main container (where the project env vars are present). Can you expand a little more on why it's not possible to use the Datadog Agent Orb setup step in your workflow?

In any case, we are actively working to remove the need to run the Datadog Agent to report test results, so this won't be a problem in the near future.