search

acquia / moonshot

Moonshot: Because releasing services shouldn't be a moonshot!
Apache License 2.0
52 stars 49 forks source link

CPD-183: Added ASG rotation plugin. #267

Closed AbhishekRana95 closed 4 years ago

AbhishekRana95 commented 4 years ago

Change: minor Purpose: feature

Manual review steps:

  1. For pointing the changes for local environment of moonshot gem: How to specify local Ruby gems in your Gemfile
  2. Once the above step is up and working, include the plugin in cloud-database-worker repo in Moonfile.rb by adding c.plugins << Moonshot::Plugins::RotateAsgInstances.new to the file.
  3. Update the AMIMap key in cdb-worker.json and change the AMIs.
  4. Run cd cloud-database-worker && bundle exec moonshot update
  5. Open the aws console and go to CloudFormation > Stacks > cdb-worker-dev-#{stack_name} where stack_name is your worker stack.
  6. Go to the ASG of the stack and check the old instances are removed and new instances are attached with the updated launch configuration for ASG.

We should also verify that the outdated instances currently in the ASG are stopped correctly.

AbhishekRana95 commented 4 years ago

@borsothy I have addressed all your comments. Please let me know if anything else needs to be changed.

AbhishekRana95 commented 4 years ago

I was looking around and found https://github.com/acquia/moonshot/blob/ab44b072e82747eccddb59b6afc081b4ae5524d7/docs/plugins/asg_rollout.md. I could not dig deeper, but can you take a look that plugin and check if we are not duplicating the same behaviour or if we can extend it's behaviour.

This asg_rollout is introducing a new instance in ASG first and then removing the old instance. Let me see if I am able to reuse this and if not then find the reason why. Thank you for pointing this out @kaushik 💯

AbhishekRana95 commented 4 years ago

I was looking around and found https://github.com/acquia/moonshot/blob/ab44b072e82747eccddb59b6afc081b4ae5524d7/docs/plugins/asg_rollout.md. I could not dig deeper, but can you take a look that plugin and check if we are not duplicating the same behaviour or if we can extend it's behaviour.

This asg_rollout is introducing a new instance in ASG first and then removing the old instance. Let me see if I am able to reuse this and if not then find the reason why. Thank you for pointing this out @kaushik 💯

@kaushik there is a case in this that suppose there is no outdated instances in the ASG at the moment and we run bundle exec moonshot update in the repo. What happens is it first increase the desired capacity by 1, wait for and add an instance and then start the logic where it finds the outdated instances if not then what it does is that it decreases the capacity by 1 i.e. getting it back to normal. When the capacity is brought back to normal what AWS does is it terminates one of the instance in respond to the change and this termination is not doing it in the way we want and doing it according to the AWS termination logic which might lead to abruptly stopping the instance which might be executing some workflow at the time.

Let me know your thoughts on this 🤔

glofishh commented 4 years ago

Manually reviewed! Verified instance was properly stopped, removed, and rotated.

Console output

|gloria.lee| |cloudservicesdev| (master)$ [2020/10/12-16:22:03]\ in ~/Desktop/Database/cloud-database-worker
  → bundle exec moonshot update -n dev-gloria1
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.
[ ✓ ] [ 0m 12s ] ChangeSet moonshot-cdb-worker-dev-gloria1-1602534307 ready!
* Modify LaunchConfig (AWS::AutoScaling::LaunchConfiguration)
 - Will be replaced
 - Caused by template change (Properties: ImageId)
* Modify WorkerASG (AWS::AutoScaling::AutoScalingGroup)
 - May be replaced (Conditional)
 - Caused by LaunchConfig (ResourceReference)
Apply changes? yes
[ ✓ ] [ 0m 0s ] Executed ChangeSet moonshot-cdb-worker-dev-gloria1-1602534307 for CloudFormation Stack cdb-worker-dev-gloria1.
[ ✓ ] [ 0m 22s ] CloudFormation Stack cdb-worker-dev-gloria1 successfully updated.
[ ✓ ] [ 0m 1s ] CodeDeploy Application cdb-worker-dev-gloria1 already exists.
[ ✓ ] [ 0m 0s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-gloria1 already exists.
[ ✓ ] [ 0m 0s ] AutoScaling Group(s) up to capacity!
[ ✓ ] [ 0m 1s ] Uploading 'cdb-worker_1602534353_gloria.lee.tar.gz' to 'cdb-stack-resource-backups-dev' succeeded.
[ ✓ ] [ 4m 25s ] ASG instances rotated successfully!
[ ✓ ] [ 0m 2s ] Outdated instances removed successfully!

In AWS console

Before update

asg_new

After update Outdated instance successfully shut down:

asg_deleted

New instance created/rotation completed:

asg_new