[ECS] Run ECS task definitions locally

[clareliguori]

Be able to run an ECS task definition for local development. Test your application code in a local container against a container credentials endpoint, a task metadata endpoint, and task definition attributes like environment variables and secrets.

[deleugpn]

At the moment I work with about 30 containers that runs on Fargate and for local development we just use docker-compose. I don't feel like I'm missing out on anything. However, if this comes out easier to replicate than docker-compose by allowing us to export an entire cluster and easily spawn that locally, then I could get interested in the switch. Container networking is a must have, imo.

[clareliguori]

@deleugpn What do you mean by container networking? What networking features/behavior do you need to simulate locally?

[deleugpn]

The same provided by docker compose. If I turn on 10 task definitions, they need to be able to communicate with each other through a DNS. On ECS we have Service Discovery that uses Route 53 to accomplish that. Docker compose make fake dns using the service name. Whatever solution this yields, service communication will likely be a must have.

One gotcha that just occurred to me: please consider the entire application stack. Just running my ECS cluster locally isn't enough, I also need to simulate my RDS by starting a database container. I might need a load balancer (nginx-proxy). If I need DynamoDb or Sqs, we have localstack. When considering all this, I'm actually realizing the value Docker compose provides: we have all the same images we pushed to ECR + networking + extra containers (database, nginx, redis, localstack). It's a pretty powerful reproduction.

From my perspective (which might not be your target audience), this would need: A) make it easy to run my entire cluster locally B) make it easy for me to attach AWS replacement resources to my local cluster. (redis, database, nginx, etc)

Am I making sense?

[lsegal]

Even a tool that could take docker-compose (or similar tool) configuration to generate tasks and vice versa would go a long way to lowering the bar for local development, or at the very least help validate ECS infrastructure prior to full-scale deployment.

[clareliguori]

@lsegal have you seen the support in ecs-cli for running tasks from docker-compose files? It is missing the opposite though: task def to docker-compose

[dontrebootme]

Base images that use private registry like ecr when run online but probably won't use ecr for local (due to local users vs instance/task role). I guess that's summed up to local with to ecr for base images.

Some pain points for local vs ECS today Environment variables Secrets Logging (awslogs vs local) Cloudwatch? Load balancing (x-forwarded-for) vs local deploy Certificates

[talawahtech]

Off the top of my head, loading configuration data from the Parameter Store and simulating an ALB/NLB are the first things that come to mind.

[clareliguori]

@talawahtech What characteristics of the load balancer do you find you need to simulate locally?

[clareliguori]

@dontrebootme I'm having trouble parsing your comment about local images vs ECR. Do you mean your local users aren't able to pull from ECR for some reason? Or you would like to use local images when running locally, but use ECR when running in ECS?

[talawahtech]

@talawahtech What characteristics of the load balancer do you find you need to simulate locally?

@clareliguori Primarily to handle SSL and pass along the expected X-Forwarded headers

[dontrebootme]

@clareliguori for all the developers to access ecr directly would require a lot of accounts and iam credentials. We use federated logins but if there was a local story for ECS and we wanted it to be similar to ECS in AWS, I'd imagine we'd want a good story for ECR as well. Not sure the exact solution but just raising awareness that the ECS and ecr integration may be something to keep in mind when solving local ECS

[clareliguori]

@dontrebootme Got it. ECR does support federated logins on the local command line, but understood that devs would not necessarily have access to ECR for local development. We're currently planning to enable using local images and Dockerfiles instead of the images listed in the task definition. https://aws.amazon.com/blogs/compute/using-federation-with-amazon-ecr/

[lsegal]

@lsegal have you seen the support in ecs-cli for running tasks from docker-compose files? It is missing the opposite though: task def to docker-compose

@clareliguori ecs-cli is great for getting up and running once everything is in place, but it doesn't spit out the task definitions locally, so it's difficult to understand what even a simple docker-compose.yml file will actually yield in production. In my experience, a lot of time is wasted tweaking settings on live instances to work around these subtle differences-- there doesn't seem to be a way around pushing directly to "prod" (prod being AWS here) and debugging the values afterwards, unless I'm missing something.

A local ECS runner could give you a better idea of what your docker-compose will give you, but in absence of running code locally, at least having a dry-run mode in ecs-cli to give you the expected container, service, and task definition JSON/CF templates/etc, prior to deploying could help quite a bit in validation. I think aws/amazon-ecs-cli#588 could kind of cover this (if it printed the expected inputs).

[clareliguori]

We launched the ECS Local Container Endpoints project today, which represents a big step towards being able to run your ECS tasks locally. Check it out, and please give us feedback! https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/

Still to come: Being able to take a task definition file and run that locally, including environment variables and secrets.

[coultn]

Launched! https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-ecs-now-offers-improved-capabilities-for-local-testing/