new example: demonstrate basic ASG integration with Lifecycle Hooks

[ketzacoatl]

Autoscaling Groups can be setup with one or more Lifecycle hooks, which can then be used to take specific actions as EC2 instances pass through the various states of their "lifecycle". In essence, lifecycle hooks put EC2 instances in a "paused/pending" state before going into service, and before terminating. However, implementing the full picture requires linking together several components, and there are some options for how it's all setup.

The goal of this ticket is to produce an initial example that we can use to explore specific use cases and implementation ideas when in those scenarios. Future tickets can add to this example or create additional variations on the example to cover other use cases.

We're already using user_data for init (though lifecycle hooks could still bring improvements there), so at least right now, we're primarily interested in taking actions before terminating an EC2 instance.

Specifically, we'd like build out coverage for:

• de-registering EC2 instances from an ELB or ALB's target group before terminating each instance
• when using a cluster management tool (Kubernetes or Nomad, for example), we should use the manager's APIs to mark the instance to terminate as in-eligible for new tasks and should also be drained of currently running tasks.

We are interested in two main ways to run actions on lifecycle events:

• a lambda function
• some script or tool that runs from an EC2 instances

Interacting with the AWS API is a pretty easy thing to do with lambda, but using tools like kubectl to interact with cluster management APIs is not. Due to our use of cluster management tools, running the tool/script from an EC2 instance is more useful and desirable than running from lambda. However, we would like to demonstrate how to use lambda for running actions on lifecycle events.

Requirements

• Must use an existing example which has an ALB and some webapp servers in an ASG.
• Could demo how to run a "cleanup action" on a EC2 webapp instance when it is in the terminated lifecycle hook - that is, each instance would run an action on itself when it is terminating.
• Should add an EC2 instance that can be used to run actions on lifecycle events (let's call it the "Monitor")
• Must give that EC2 instance the permissions it needs (to access resources on AWS) via IAM instance profile
• Should a tool/script/agent (lifecycled or some other demo script), on the Monitor, to handle lifecycle events for all the webapp servers. Do something like de-register it from the ALB
• If and where it makes sense, should encapsulate lifecycle components in modules (or at least note that as a task for later).
• Should leave running actions from lambda as a task for another example, or to include in this example later, unless everything else is complete and there is time available in this issue.

Resources

• Here is more info about Lifecycle Hooks from the AWS docs: https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html
• Terraform docs: https://www.terraform.io/docs/providers/aws/r/autoscaling_lifecycle_hooks.html
  • note the overlap between this resource and using the lifecycle parameter in the ASG resource
  • ASG docs https://www.terraform.io/docs/providers/aws/r/autoscaling_group.html
• lifecycled: https://github.com/buildkite/lifecycled
• This is a simple but complete example of the general setup: https://github.com/buildkite/lifecycled/blob/master/terraform/modules/example/main.tf
• Another blog post describing some aspects of this: https://aws.amazon.com/blogs/compute/how-to-automate-container-instance-draining-in-amazon-ecs/

[ketzacoatl]

Sending to @psibi for review.