Getting and setting DD_API_KEY at runtime

[ac-hibbert]

Describe what happened:

I am looking to fill out DD_API_KEY at runtime instead of passing it in as an environment variable to keep it secure. i am using ECS Fargate and I'm using vault as a secret backend

I can replace /etc/cont-init.d/01-check-apikey.sh with a script which does an export DD_API_KEY=$(vault read command) and it carries on to do the check for DD_API_KEY as per the script in this repo. The problem is that the export command during s6overlay is not actually setting it as a runtime variable, so the container is not actually forwarding metrics.

I have also tried adding "ENV S6_KEEP_ENV=0" and changing /etc/cont-init.d/01-check-apikey.sh to have shebang "#!/usr/bin/with-contenv sh", but then it fails to start the agent

[services.d] starting services
[ AGENT ] starting agent
[ AGENT ] fdmove: fatal: unable to exec agent: No such file or directory
[ TRACE ] starting trace-agent
[ TRACE ] fdmove: fatal: unable to exec trace-agent: No such file or directory
[PROCESS] starting process-agent
[PROCESS] fdmove: fatal: unable to exec process-agent: No such file or directory
[ AGENT ] AGENT EXITED WITH CODE 111, SIGNAL 0, KILLING CONTAINER
[ TRACE ] trace-agent exited with code 111, signal 0, restarting in 2 seconds
[PROCESS] trace-agent exited with code 111, signal 0, restarting in 2 seconds
[services.d] done.

Is there a way of getting around this? Or a more recommended way to keep it secure.

Additional environment details (Operating System, Cloud provider, etc):

AWS ECS Fargate

[danelowe]

Did you find a solution to this @hibbert?

I've been stuck on this for a few hours and am getting nowhere.

I've tried bypassing this environment variable check, and adding the api_key value directly to /conf.d/datadog.yaml and /etc/datadog/datadog.yaml. I still get ERROR (file.go:34) - you must specify an API Key, either via a configuration file or the DD_API_KEY env var

I've tried setting the DD_API_KEY env var directly in /etc/cont-init.d/01-check-apikey.sh with the same result.

I've tried mounting the secret file to /var/run/s6/container_environment/DD_API_KEY and it doesn't seem to do anything. I've tried researching s6, and it seems so complicated that I'd probably spend a week on it and still not know what it is even meant to do.

I can't find a way of running DataDog agent docker container without setting the API key directly in the stack's config.

[danelowe]

Ah, So all I needed was a bit of a rest, and going back to basics.

Clearly, if the Environment Variable is present when the entrypoint/command is called, it works.

So rather than trying to understand how all this stuff works, I just needed to wrap the entire entrypoint.

E.g. to test without creating a new image:

./tools/dd_fileenv.sh

#!/usr/bin/env bash

file_env() {
    local var="$1"
    local fileVar="${var}_FILE"
    local def="${2:-}"
    if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
        echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
        exit 1
    fi
    local val="$def"
    if [ "${!var:-}" ]; then
        val="${!var}"
    elif [ "${!fileVar:-}" ]; then
        val="$(< "${!fileVar}")"
    fi
    export "$var"="$val"
    unset "$fileVar"
}

file_env 'DD_API_KEY'

exec "$@"

docker-compose.yml

...
  apm:
    image: datadog/agent
    entrypoint: ["bash", "/dd_fileenv.sh"]
    command: ["/init"]
    volumes:
      ...
      - ./tools/dd_fileenv.sh:/dd_fileenv.sh
      - ./tools/dd_api_key:/run/secrets/dd_api_key
    environment:
      DD_API_KEY_FILE: /run/secrets/dd_api_key
...

For production deployment, I think I'm just going to store the API Key directly in a bash script that exports the environment variable and then calls the argument (command). That bash script will be stored in docker secrets.

[carlosmmelo]

I'm getting this using today's latest image, it runs for a while and then all of the sudden the container loses all the Env Var that were already set when starting to container