aws-samples / aws-codedeploy-samples

Samples and template scenarios for AWS CodeDeploy
Apache License 2.0
639 stars 780 forks source link

Blue/Green Deployments with manual rerouting and scaling events in between Ready -> ReRoute #80

Open knizami opened 6 years ago

knizami commented 6 years ago

Hello, in #66 it is indicated that these scripts are not required for Blue/Green. However, I think they may be? In our situation we are doing a blue/green deployment, we initiate a codedeploy deployment and the new revision is installed onto the replacement instances (an autoscaling group) and reaches a ready state ready to reroute traffic.

After this, there is validation that takes place on the replacement instances outside of codedeploy. However, if there is a scaling event on this replacement autoscaling group, then the new instances that come up will not get a deployment (because there is no codedeploy lifecycle hook on this replacement asg). Once the complete / reroute action is initiated on CodeDeploy then traffic is rerouted to the instances that were present in the asg when the deployment began. The other instance that resulted from a scaling event will just sit in the ASG with no traffic routed to it.

There is a more destructive scenario, instead of a scaleout, a scalein occurs after the deployment is in a ready state. Now, the instance has been terminated but codedeploy doesn't know this yet in the deployment. Once a complete / reroute action is taken then the deployment fails when it tries to route traffic to the terminated instance.

One solution seems to be to turn off alarms at ApplicationStop and then turn them back on AfterAllowTraffic using these scripts. However, there is still a chance that a scaling event could occur between the AfterAllowTraffic <-> AfterBlockTraffic. What's the right way to handle this scenario?