[ECS] [request] Passing container instance environment variables to agent

[victorbordo]

Summary

I want to inject a system-level environment variable FOO exported on all cluster container instances into ECS tasks. This is a separate environment variable being used as a required identifier separate from app configuration pulled using parameter store. What's the recommended way to do this?

Description

FOO is exported on the container instance in /etc/environment. Here are a few routes I've considered:

1. Adding value to /etc/ecs/ecs.config.
2. Creating a separate configuration file in /etc/ecs with the value.
3. Passing FOO directly when the ecs-agent is started:

docker run --name ecs-agent \
    --detach=true \
    --restart=on-failure:10 \
    --volume=/var/run/docker.sock:/var/run/docker.sock \
    --volume=/var/log/ecs:/log \
    --volume=/var/lib/ecs/data:/data \
    --net=host \
    --env-file=/etc/ecs/ecs.config \
    --env=ECS_LOGFILE=/log/ecs-agent.log \
    --env=ECS_DATADIR=/data/ \
    --env=ECS_ENABLE_TASK_IAM_ROLE=true \
    --env=ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true \
    --env=FOO=${FOO} # PASSING NEW ENV VAR
    amazon/amazon-ecs-agent:latest

Expected Behavior

Each task in the ECS service has access to FOO as an environment variable.

Observed Behavior

With the current configuration, FOO is available on all container instances shell environments but isn't passed through to tasks. This is expected because the ecs-agent is isolated from the host environment.

Environment Details

Docker version: Docker version 19.03.13-ce, build 4484c46 Running ECS Optimized Image

Supporting Log Snippets

I can't post logs here.

[fenxiong]

Hi, ECS supports passing environment variable as a file to containers: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/taskdef-envfiles.html. To use that feature for your case, I think you can put what you have in /etc/environment to a file on an S3 bucket and then reference the file in the task definition, and then your container should have access to the environment variables specified in the file. Does that satisfies your use case?

[victorbordo]

Thanks for responding so quickly @fenxiong. If FOO was a single static value shared across task run contexts this would work. Unfortunately I don't think this satisfies my use case because the value FOO is a unique identifier that's set dynamically based on the total number of container instances in the cluster.

[rgleme]

This would help an issue like this: https://github.com/aws/containers-roadmap/issues/1266

[fenxiong]

If FOO was a single static value shared across task run contexts this would work. Unfortunately I don't think this satisfies my use case because the value FOO is a unique identifier that's set dynamically based on the total number of container instances in the cluster.

i see. this use case is something we don't natively support right now, and there have been a few feature requests that have similar asks:

• https://github.com/aws/containers-roadmap/issues/369
• https://github.com/aws/containers-roadmap/issues/1485

so i'm transferring the issue to container roadmap to track this as a feature request as well.

[victorbordo]

Thanks for transferring to the roadmap. Any suggestions on workarounds?

[fenxiong]

One workaround i can think of is to mount the host /etc/environment file to the container with a bind mount [1], and then load the envs when the container starts (e.g. [2]). For example, i was able to use the following task definition that let the container to get the environment variable from the host /etc/environment file:

{
  "family": "test-gh3075",
  "containerDefinitions": [
    {
      "name": "test",
      "cpu": 10,
      "memory": 64,
      "image": "public.ecr.aws/x1v8g9p0/busybox:latest",
      "command": [
        "sh",
        "-c",
        "export $(cat /etc/environment | sed 's/#.*//g' | xargs); echo $FOO"
      ],
      "mountPoints": [
        {
          "containerPath": "/etc/environment",
          "sourceVolume": "env-mount"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "gh3075-test",
          "awslogs-stream-prefix": "test1",
          "awslogs-region": "us-west-2",
          "awslogs-create-group": "true"
        }
      }
    }
  ],
  "volumes" : [
    {
      "host" : {
        "sourcePath" : "/etc/environment"
      },
      "name" : "env-mount"
    }
  ]
}

(and echo $FOO can be changed to whatever your container actually does).

@victorbordo does the above workaround works for you?

[1] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bind-mounts.html [2] https://gist.github.com/mihow/9c7f559807069a03e302605691f85572

[victorbordo]

@fenxiong this is a valid way to go. I ended up using a different solution that involves pulling the configuration I need from container instance tags using the ecs metadata endpoint. Thanks again for your suggestions and triaging the issue. Hope to see this feature end up in a future release.