deitch / aws-asg-roller

Manage rolling upgrades for AWS autoscaling groups
Apache License 2.0
57 stars 17 forks source link

Adds support to store original desired count as an ASG tag #37

Closed outofcoffee closed 4 years ago

outofcoffee commented 4 years ago

This adds support to store the 'original desired count' as an ASG tag. This is important when the roller process terminates, such as during maintenance (or instance rolling!).

Currently, if the roller has increased the desired count of an ASG (e.g. 3->4), then restarts, the next loop will treat the increased value (4) as if it was the original desired count and increase it again (4->5). This leads to mis-sized ASGs.

Implementation notes

This behaviour is enabled by default, per review notes. The list of IAM permissions in the README has been updated to include autoscaling:CreateOrUpdateTags.

~This behaviour is controlled by an optional environment variable: ROLLER_ORIGINAL_DESIRED_ON_TAG. This is disabled by default to preserve current semantics.~

~If ROLLER_ORIGINAL_DESIRED_ON_TAG is set to true then,~ the roller examines the ASG for the aws-asg-roller/OriginalDesired tag. If this is not present, the current DesiredCapacity is used and stored in the tag. Future loops will use the value of the tag instead of the current DesiredCapacity to ensure that the correct size is restored.

Logs

The (verbose only) logs for this behaviour are as follows:

When there is no tag value:

2020/03/27 11:44:02 guessed desired value of 4 from current desired on ASG: my_asg_name

When writing the tag value:

2020/03/27 11:44:02 recorded original desired of 4 in tag on ASG: my_asg_name

When reading the tag value:

2020/03/27 11:44:02 read original desired of 4 from tag on ASG: my_asg_name