Better instance replacement

[k2nr]

The current rolling update has been working fine but it has some problems. ECS released task constraint recently http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_PlacementConstraint.html with task constraints and autoscaling group lifecycle hook, we can do instance replacement better. Here's my rough idea:

Prerequisites

• Set CloudFormation's Autoscaling UpdatePolicy to "replacement"
• Add "not equal to draining" constraint to all barcelona ECS services

Replacement Strategy

2. When ASG replacement is triggered, LAUNCHING lifecycle hook is executed and the hook adds "draining" label to all existing container instances
3. After new ASG and its instances start, TERMINATING lifecycle hook is executed and the hook restarts all ECS services
   1. Then new services will be placed new container instances because exisisting instances have "draining" label and thus tasks cannot be placed
   2. When all services are deployed, the old instances do not have ECS tasks running
4. After the step 3 finishes, the old instances start shutting down

Benefits

Compared to the current "rolling update" strategy, this way has several benefits

• Faster
• When replacement fails (before "service deployment" in the step 3), it's really safe to rollback
  • In the step 3, instance replacement already finished so there's no need for rolling back even if something wrong happens

[k2nr]

ECS has released DRAINING state feature (I had been waiting this feature for a year!), so we no longer need to implement this complex replacement logic. I'll rewrite the description

[k2nr]

This is done