Continuous delivery?

[bsideup]

Images are hardcoded in the example.

In a real world scenario, they will be changed over the time.

But because of https://forums.aws.amazon.com/thread.jspa?messageID=741932#741932 it's impossible to define the stack without the hardcoded images.

Could you please update an example to demonstrate how to do CD? Otherwise, it's a bit useless :)

Thanks!

[PaulMaddox]

Hi @bsideup,

Typically this would be done by having your CI/CD process modify (or regenerate) the TaskDefinition within services/*/service.yaml to use the correct image/version, and then run a CloudFormation update-stack operation. The CloudFormation update will then roll out the new/updated task definition across your cluster.

If you are using Jenkins, there is a plugin for deploying CloudFormation stacks: https://wiki.jenkins-ci.org/display/JENKINS/AWS+Cloudformation+Plugin

Your link to the forum post does raise a valid issue though; In their current configuration the task definitions in this repository will be duplicated on change (rather than the revision incremented). On Sept 19th, we announced CloudFormation support for the family parameter which should make this a lot nicer/easier, however we are aware of an issue with it's implementation that is preventing it from working as it should. I have spoken to the CloudFormation and ECS teams regarding this, and they are working on a fix. I will keep this GitHub issue open until that fix is released.

A concrete end-to-end example of how to do continuous delivery with ECS is something we are looking to publish in the near future so watch this space.

[bsideup]

@PaulMaddox cool, thanks!

[cloudoperator]

That would be great. Trying to get Codeship + Bitbucket + ECR + ECS is my current goal! Unless there is something better.

[PaulMaddox]

Hi @bsideup,

Good news - the CloudFormation/ECS teams have resolved the issues that caused task definitions to not be deployed correctly when specifying the family property. I have tested this and can confirm that when deploying that by adding the family property to the TaskDefintion, all future updates to the definition are deployed as a new TaskDefinition revision (rather than a duplicated definition).

If you're coming to our re:Invent conference next week I would recommend checking out some of the ECS sessions we've got planned:

CON301 – Operations Management with Amazon ECS CON302 – Development Workflow with Docker and Amazon ECS CON303 – Introduction to Container Management on AWS CON308 – Service Integration Delivery and Automation Using Amazon ECS CON309 – Running Microservices on Amazon ECS CON310 – Running Batch Jobs on Amazon ECS CON311 – Operations Automation and Infrastructure Management with Amazon ECS CON312 – Deploying Scalable SAP Hybris Clusters using Docker CON313 – Netflix: Container Scheduling, Execution, and Integration with AWS CON316 – State of the Union: Containers CON401 – Amazon ECR Deep Dive on Image Optimization CON402 – Securing Container-Based Applications

There are also two hands-on workshops: CON314 – Workshop: Build a Recommendation Engine on Amazon ECS CON315 – Workshop: Deploy a Swift Web Application on Amazon ECS

In particular, CON302 and CON309 should really help with the CI/CD workflow. If you can't make it - not to worry, these will be recorded and uploaded to our YouTube channel shortly afterwards.

[PaulMaddox]

Just to follow up on this... we've now published a reference delivery pipeline for ECR/ECS using CodePipeline and CodeBuild. This is a great pattern for deployment right from source repo to your ECS cluster(s).

https://github.com/awslabs/ecs-refarch-continuous-deployment

[davidham]

@PaulMaddox I'm using this project's architecture for my EC2/ECS/CFN setup, and I'm trying to adapt the deployment-pipeline.yaml from the ecs-refarch-continuous-deployment project into this architecture. It's going OK, but your setup is a little different from theirs and I'm not quite understanding yet.

In the ecs-refarch-cloudformation setup (if I follow it correctly), each service registers a target group and a listener rule with the ALB, and the ALB routes traffic to each service based on if the traffic matches the listener rule.

In my setup, I have a handful of services, but the path matching won't work for me, so I set up a different listener for each service and gave each one its own port. So far so good. I have two services up and running, each on its own port.

I'm also deploying these stacks separately. I have one stack that handles the VPC, security groups, ALB, and cluster, and I want to deploy the services and their build pipelines separately.

I have the build pipeline working up until the deploy step. When it updates my service stack, it seems to be recreating all the assets, like the target group and listener rule. I don't think I want that; I just want it to update the task definition and then point the service at it. Can I do this without detaching and reattaching from the ALB, and recreating all the service's resources?

[jpignata]

You should certainly be able to update the stack in the CodePipeline deploy stage. Is it creating an entirely new stack rather than running an update?

[davidham]

Yes, when I finally figured it out, that's what it was doing. I had passed the wrong service stack name into my CodePipeline stack, so when it got to the deploy stage, it was trying to spin up a whole new service. Once I got that sorted it works like a champ. Thanks for the reply!