aws-samples / aws-codedeploy-samples

Samples and template scenarios for AWS CodeDeploy
Apache License 2.0
639 stars 780 forks source link

Why does this change ASG desired and min? (question) #57

Closed ghost closed 7 years ago

ghost commented 8 years ago

I'm using this and I noticed that this code tends to change my desired and min values from 1 each to 0. If my code deploy fails, it never reverts those back and I run the risk of ASG spinning everything down and my site being down. If I set my min and desired values, those should remain set as defined, not changed arbitrarily to pull an instance out. Are there really no commands to deregister a system specifically that it has to go through changing these values?

schmohlio commented 7 years ago

+1 for this. Why do we need to do this if we are turning off autoscaling beforehand?

mmerkes commented 7 years ago

@j88per I see that this comment is pretty old and nobody has responded yet. I'm going to make some assumptions based off of guessing the intention of the author of the script :)

I'm not sure what script specifically you're using, but I assume you're talking about the behavior that happens here. The script is assuming that you have more than one host in the ASG. The scripts puts the host into StandBy during the deployment, and if the host is in StandBy, I believe ASG will try to replace it with a new host if you don't reduce the min/desired values. Regardless, if you only have one host, won't that mean that your website will be out for the duration of your deployment? I would recommend having at least two hosts for your system to avoid being out. However, if you don't mind the brief outage and want to stop traffic from going to your host, you could remove this if statement from your script, and it looks like it will deregister the host from the load balancer directly without changing your ASG settings, though if it fails an ASG health check in the process, it might replace your host.

If ASG spins up a new host before you complete your deployment, it will try to deploy the last successful revision, which would be one revision behind and why this script tries to avoid that situation.

I believe this answers your question, so I'm going to close this issue. Feel free to add additional comments or re-open the ticket.

ghost commented 7 years ago

At the time I wrote this, I was setting up a new auto scaling setup, most of the year we run a single node as it's not very active. 2 or 3 times out of the year, for a few days, we run live auctions and wish to scale up/down based on participation. So having 1 node is fine for part of the year.

However, occasionally I noticed that my values for desired servers and min servers from whatever values I defined to 0. It seems that the elb register/deregister scripts which I do run, would modify my settings in the console. This bothers me. Especially if it's just doing something simple like $val-1 and eventually getting to 0.
What happens next is that ELB see's desired/min 0, and doesn't spin up any servers because the scripts modified the value. So basically, I have a desired and min value, and the elb deregister script changes my value to 0. Why? that's really my question.

Not oh hey, here's a bug, but if I set a value in desired and/or min - why is a script even changing those? I understand that spinning up or down servers affects how many are running as well as logic to determine if nodes should be spun up, but why would a script modify a setting as critical as that?

mmerkes commented 7 years ago

I understand your concerns. It's definitely not ideal to be touching people's min size as it's kind of a hacky way to get the desired outcome and obviously has limitations. This is a one size fits all bash script, and it definitely doesn't handle your use case well. I opened up issue #70, which I think captures the heart of the issue, though doesn't necessarily provide a clear solution. If you use classic ELB, I'd consider onboarding with CodeDeploy ELB support, which should be much more reliable. Otherwise, I'd edit your script accordingly to get the behavior you need. Changing the 0 to a 1 in this line should at least keep it from going to 0.