aws: Allow rolling updates for ASGs

[radeksimko]

Once #1109 is fixed, I'd like to be able to use Terraform to actually roll out the updated launch configuration and do it carefully.

Whoever decides to not roll out the update and change the LC association only should still be allowed to do so.

Here's an example from CloudFormation: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatepolicy.html

How about something like this?

resource "aws_autoscaling_group" "test" {
  rolling_update_policy {
    max_batch_size = 1
    min_instances_in_service = 2
    pause_time = "PT0S"
    suspend_processes = ["launch", "terminate"]
    wait_on_resource_signals = true
  }
}

then if there's such policy defined, TF can use autoscaling API and shut down each EC2 instance separately and let ASG spin up a new one with an updated LC.

[pmoust]

:+1:

Our use case pretty much depends on this to be crystal clean. Other wise rolling updates require some minimal external intervention which I am working on making obsolete.

Huge +1 on this

[matthewford]

:+1:

[jessem]

:+1: Likewise. Using a python script to handle this, which makes it a bit clunky to keep things simple with terraform.

[chrisferry]

:+1:

[phinze]

@radeksimko totally agreed that this is desirable behavior.

It's worth noting, though, that this is CloudFormation-specific behavior that's not exposed up at the AutoScaling API. I think the way we'd be able to achieve something similar would be to build some resources based on CodeDeploy:

https://docs.aws.amazon.com/codedeploy/latest/APIReference/Welcome.html

[johnrengelman]

My experience with code deploy is that it's limited to installing software on running instances...so it can roll out updates to ASG instances but it doesn't know how to do a roll-out by 'terminating' instances like CF does. This would be an awesome feature, but I don't really think it's something that fits into Terraform's current model very well. Perhaps if there was a separate lifecycle type hook that actions could be plugged into.

So something like:

resource 'aws_launch_configuration' 'main' {
}

resource 'aws_autoscaling_group' 'main' {
  lifecycle {
    on_update {
      properties ['launch_configuration']
      actions {
        action 'aws_autoscaling_group_terminate_instances' {
          batch_size = 1
        }
      }
    }
  }
}

These actions then could be a whole separate concept in Terraform.

[woodhull]

:+1: just realized this. did not realize that this was implemented by AWS as a cloudformation primitive rather than an ASG primitive we could hook into.

[woodhull]

Is anyone experimenting with using other tools to hack around this terraform limitation? Even if we were to combine terraform with external tooling like http://docs.ansible.com/ec2_asg_module.html I'm not sure where we'd hook in. A local-exec provisioner seems like the right thing -- but those are only triggered on creation, not modification. Maybe as a placeholder before implementing a native rolling update solution terraform could offer some sort of hook for launch configuration changes that we could use to trigger an external process?

Otherwise, I think we'll need to manage AMI version updates externally via ansible or some homegrown tool and then use terraform refresh to pull them in before doing plan or apply runs. It's all starting to drift away from the single-command infrastructure creation and mutation dream we had while starting to use terraform.

As part of our migration to terraform/packer away from a set of legacy tools we had been planning a deployment model based on updating pre-baked AMIs created with packer on a rolling basis. Any other ideas for workarounds that we could use until terraform is able to do this sort of thing out of the box?

[johnrengelman]

I've been using bash around the AWS cli. It would be awesome to implement tasks in Go against the awslabs library and then just call them from terraform though.

[woodhull]

Over the weekend I wrote some code to handle blue/green deploys for our particular use case in ruby, adding to our ever growing wrapper of custom code needed to actually make terraform useful.

Rather than hooking into terraform events it's ended up as a huge hack: it dynamically modifies the terraform plan to do the create of the new ASG with new AMI alongside the existing ASG. Then it uses ruby to make sure the new ASG is up and working correctly before removing the old ASG, regenerates the terraform plan file to match the new state, and finally calls terraform refresh so that the tfstate matches the new reality we've created.

Would be great if this sort of workflow for mutating a running app when we've updated an AMI were built in or if there were at least easy ways to hook into terraform to add custom behavior like this beyond learning Go and Terraform internals. In our case, even if terraform could handle the ASG operations for us, we'd still like to be able to run our quick, custom sanity check script to make sure everything is working properly on the new instances before removing the old ASG from the pool.

[woodhull]

This feature might be the most straightforward (although awkwardly round-about) way to get rolling deploys working inside terraform: https://github.com/hashicorp/terraform/issues/1083

[ketzacoatl]

+1.. this would make a huge difference in some of my current workflows

[d4v3y0rk]

+1

[ajmath]

+1

[blewa]

+1

[nathanielks]

@woodhull that wrapper wouldn't be around somewhere we could take a looksee, would it? :smile:

[woodhull]

@nathanielks I copy/pasted some fragments of the code here: https://gist.github.com/woodhull/c56cbd0a68cb9b3fd1f4

It's wasn't designed for sharing or reusability, sorry about that! I hope it's enough of a code fragment to help.

[nathanielks]

@woodhull you're a gentlemen, thanks for sharing!

[BrunoBonacci]

+1

[saliceti]

+100

[ryan0x44]

If anyone is trying to achieve Blue-Green deployments with AWS Auto Scaling and Terraform, I've made a note about how I did it here. Not sure if it's an ideal solution, but it's working for me and proving to be very reliable at the moment :)

[nathanielks]

Thanks @ryandjurovich!

[ernoaapa]

:+1:

[farridav]

+1 just about to look at how to do this, would love to avoid writing a script to down and up instances to use the new Launch Configuration will take a look at @ryandjurovich's script also :)

[robmorgan]

:+1:

[StrongPa55word]

+1

[farridav]

I'm sure there is a genuine need for rolling updates to an ASG, so wont detract from this too much, but anyone reading this issue that is OK with blue/green deployment as opposed to rolling updates, I've done so successfully and cleanly by using Terraform built in functionality, as suggested in this really useful post apparently its how HashiCorp do their production rollouts :)

@ryan0x44 I'm afraid your gist link is dead

[BrunoBonacci]

Hi @farridav,

Blue/Green deployments are a better solution for stateless services (such as: web servers, web services), but for stateful services (such as: Databases, SearchIndexes, Caches, Queues) it is often not possible or not advisable to do so. Immagine a 10 nodes db cluster with 10TB of data, spinning up a complete new cluster will cause the full resync of the 10TB of data from the old cluster to the new cluster all at once, this might saturate the network link and cause denial of service, where maybe all you wanted was to increase the number of connections. And the size of the data is not the only problem. Clusters which support ring or grid topologies are have less problems to grow/shrink elastically, but master/slave topologies are more complicated as you can grow only slaves and you have to have a procedure to step down the master and elect a slave. The possibility to do a controlled rolling update, in these type of situations is far simpler and has less impact (eg: rolling only one data node at the time, wait for data resync, then move to the next).

[StrongPa55word]

@farridav as per the post https://groups.google.com/forum/#!msg/terraform-tool/7Gdhv1OAc80/iNQ93riiLwAJ

Everything seems to be working as expected 1 - Updating the ami creates new launch config 2 - Updates the autoscaling to new launch config name However new machines are not launching automatically using the new LC/ASG created . What could be the issue here ?

[jedineeper]

@titotp check the activity history tab on the ASG, should tell you why its unable to launch the instances.

[StrongPa55word]

I am not seeing any activity on the autoscaling group

[ryan0x44]

@farridav - the link should be working now, sorry!

[ckelner]

This looks like a wonderful feature! :+1:

[igoratencompass]

+1

[rbowlby]

+1

I'd enjoy the simplicity of immutable infra. Moving away from deploying new artifacts to existing instances, and instead just phasing in new instances/containers. Without a terraform construct to do so it is a hard sell.

Additionally, having to hack together a solution that requires external scripting and/or more than one apply limits the ability to make use of Atlas. I would enjoy throwing out ansible deploy playbooks, aws code-deploy, fab, and all the complexity they introduce.

[larsp]

+1

[utobi]

+1

[avengerine]

+1

[bigkraig]

:+1:

[chainlink]

:+1:

[serbaut]

+1

[robmorgan]

I wrote a blog post based on Paul Hinze's example: http://robmorgan.id.au/post/139602239473/rolling-deploys-on-aws-using-terraform

[lswith]

+1

[kwoods]

+1

[anosulchik]

+1

[brikis98]

I tried out the Hashicorp rolling deployment approach and while it works well with a fixed-size ASG, it doesn't work with one that is dynamically sized (e.g. in response to increased load). I didn't want to write my own hacky rolling deployment script and decided to see if I could somehow leverage the rolling deployments provided by the CloudFormation UpdatePolicy.

To make it work, I defined everything as usual in my Terraform templates, except for the Auto Scaling Group, which I defined using CloudFormation in a Terraform aws_cloudformation_stack resource:

resource "aws_cloudformation_stack" "autoscaling_group" {
  name = "my-asg"
  template_body = <<EOF
{
  "Resources": {
    "MyAsg": {
      "Type": "AWS::AutoScaling::AutoScalingGroup",
      "Properties": {
        "AvailabilityZones": ["us-east-1a", "us-east-1b", "us-east-1d"],
        "LaunchConfigurationName": "${aws_launch_configuration.launch_configuration.name}",
        "MaxSize": "4",
        "MinSize": "2",
        "LoadBalancerNames": ["${aws_elb.elb.name}"],
        "TerminationPolicies": ["OldestLaunchConfiguration", "OldestInstance"],
        "HealthCheckType": "ELB"
      },
      "UpdatePolicy": {
        "AutoScalingRollingUpdate": {
          "MinInstancesInService": "2",
          "MaxBatchSize": "2",
          "PauseTime": "PT0S"
        }
      }
    }
  }
}
EOF
}

CloudFormation is, of course, more verbose than Terraform, and the plan command is not as helpful (the diff between the two CloudFormation templates is a bit hard to read). However, you can still reference Terraform resources in the CloudFormation template (e.g., note the reference to my launch configuration aws_launch_configuration.launch_configuration.name), and now, every time I update my launch configuration, I get:

1. An automatic, rolling deployment.
2. Automatic rollback if the deployment fails.
3. A log of the deployment in the CloudFormation console.

Of course, it would even better if this was built-in to Terraform (perhaps even by wrapping my solution above behind a Terraform resource, given that rolling deployments aren't exposed in the APIs), but this seems to be a passable workaround for now.

[pmoust]

@brikis98 Yeap, that's the way to go at the moment.

[maruina]

@brikis98 how do you associate an aws_autoscaling_policy to an ASG generated by Cloudformation?

The autoscaling_group_name is dynamically generated by Cloudformation itself and not exported by aws_cloudformation_stack; is something that is done manually?

[brikis98]

@maruina: You can add an output to the CloudFormation template and use it without any manual steps:

resource "aws_cloudformation_stack" "autoscaling_group" {
  name = "my-asg"
  template_body = <<EOF
{
  "Resources": {
    "MyAsg": {
      "Type": "AWS::AutoScaling::AutoScalingGroup",
      "Properties": {
        "AvailabilityZones": ["us-east-1a", "us-east-1b", "us-east-1d"],
        "LaunchConfigurationName": "${aws_launch_configuration.launch_configuration.name}",
        "MaxSize": "4",
        "MinSize": "2",
        "LoadBalancerNames": ["${aws_elb.elb.name}"],
        "TerminationPolicies": ["OldestLaunchConfiguration", "OldestInstance"],
        "HealthCheckType": "ELB"
      },
      "UpdatePolicy": {
        "AutoScalingRollingUpdate": {
          "MinInstancesInService": "2",
          "MaxBatchSize": "2",
          "PauseTime": "PT0S"
        }
      }
    }
  },
  "Outputs": {
    "AsgName": {
      "Description": "The name of the auto scaling group",
       "Value": {"Ref": "MyAsg"}
    }
  }
}
EOF
}

resource "aws_autoscaling_policy" "my_policy" {
  name = "my-policy"
  scaling_adjustment = 4
  adjustment_type = "ChangeInCapacity"
  cooldown = 300
  autoscaling_group_name = "${aws_cloudformation_stack.autoscaling_group.outputs.AsgName}"
}

Note the use of aws_cloudformation_stack.autoscaling_group.outputs.AsgName in the aws_autoscaling_policy.

[maruina]

Sweet, thank you! :+1: