search

acquia / moonshot

Moonshot: Because releasing services shouldn't be a moonshot!
Apache License 2.0
53 stars 49 forks source link

CPD-6009: Rescue and continue if the instance is already detached #274

Closed tanujjain49 closed 3 years ago

tanujjain49 commented 3 years ago

Change: minor Purpose: maintenance

Manual review steps:

  1. Get a worker stack and make sure the min instances is 2 or more else update it.
  2. Make a trivial update to launch config, updating the ami id is a good example.
  3. Issue stack update using moonshot moonshot update -n STACK_NAME
  4. while the first instance is being rotated by the rotation plugin, detach the other instance.
  5. As the plugin moves forward and tries to rotate the other instance it raises error.
  6. Update the moonshot gem to use code from this PR.
  7. Repeat steps 1 through 4. It should not raise an error
rutuja810 commented 3 years ago

Manual Review ✅ Before code changes - 

[cloudservicesdev|hosting-dev:dev-rdhimate] ~/acquia-cloud/cloud-database-worker$ bundle exec moonshot update -n dev-rdhimate
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
SIGNALFX_API_TOKEN environment variable is empty, SignalFx reporting has been disabled.
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!                                                                                 
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.                                                                   
[ ✓ ] [ 0m 9s ] ChangeSet moonshot-cdb-worker-dev-rdhimate-1629281784 ready!                                                             
* Modify LaunchConfig (AWS::AutoScaling::LaunchConfiguration)
 - Will be replaced
 - Caused by template change (Properties: ImageId)
* Modify WorkerASG (AWS::AutoScaling::AutoScalingGroup)
 - May be replaced (Conditional)
 - Caused by LaunchConfig (ResourceReference)
Apply changes? yes
[ ✓ ] [ 0m 5s ] Executed ChangeSet moonshot-cdb-worker-dev-rdhimate-1629281784 for CloudFormation Stack cdb-worker-dev-rdhimate.         
[ ✓ ] [ 0m 39s ] CloudFormation Stack cdb-worker-dev-rdhimate successfully updated.                                                      
[ ✓ ] [ 0m 4s ] CodeDeploy Application cdb-worker-dev-rdhimate already exists.                                                           
[ ✓ ] [ 0m 20s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-rdhimate already exists.                                          
[ ✓ ] [ 1m 51s ] AutoScaling Group(s) up to capacity!                                                                                    
[ ✗ ] [ 3m 36s ] Uploading 'cdb-worker_1629281997_rutuja.dhimate.tar.gz' to 'cdb-stack-resource-backups-dev' failed: Net::OpenTimeout    
[ \ ] [ 0m 8s ] Rotating ASG instances...                                                                                                
[135, 144] in /Users/rutuja.dhimate/acquia-cloud/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-28303606f170/lib/plugins/rotate_asg_instances/asg.rb
   135:       def detach_instance(instance)
   136:         @step.success("Detaching instance: #{instance.instance_id}")
   137:         require 'byebug'; byebug
   138:         # If the ASG can't be brought up to capacity, re-attach the instance.
   139:         begin
=> 140:           instance.detach(should_decrement_desired_capacity: false)
   141:           @step.success('- Waiting for the AutoScaling '\
   142:                        'Group to be up to capacity')
   143:           wait_for_capacity
   144:         rescue StandardError => e
(byebug) instance.instance_id
"i-018543b9f566c9bb4"
(byebug) continue
[ ✗ ] [ 1m 59s ] Error bringing the ASG up to capacity: The instance i-018543b9f566c9bb4 is not part of Auto Scaling group cdb-worker-dev

After code changes - -- manually detached an instance "i-0399d88e625d16e7e" from the console

[cloudservicesdev|hosting-dev:dev-rdhimate] ~/acquia-cloud/cloud-database-worker$ bundle exec moonshot update -n dev-rdhimate
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
SIGNALFX_API_TOKEN environment variable is empty, SignalFx reporting has been disabled.
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!                                                                                 
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.                                                                   
[ ✓ ] [ 0m 26s ] ChangeSet moonshot-cdb-worker-dev-rdhimate-1629282606 ready!                                                            
* Modify LaunchConfig (AWS::AutoScaling::LaunchConfiguration)
 - Will be replaced
 - Caused by template change (Properties: ImageId)
* Modify WorkerASG (AWS::AutoScaling::AutoScalingGroup)
 - May be replaced (Conditional)
 - Caused by LaunchConfig (ResourceReference)
Apply changes? yes
[ ✓ ] [ 0m 3s ] Executed ChangeSet moonshot-cdb-worker-dev-rdhimate-1629282606 for CloudFormation Stack cdb-worker-dev-rdhimate.         
[ ✓ ] [ 0m 39s ] CloudFormation Stack cdb-worker-dev-rdhimate successfully updated.                                                      
[ ✓ ] [ 0m 2s ] CodeDeploy Application cdb-worker-dev-rdhimate already exists.                                                           
[ ✓ ] [ 0m 1s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-rdhimate already exists.                                           
[ ✓ ] [ 0m 3s ] AutoScaling Group(s) up to capacity!                                                                                     
[ ✓ ] [ 0m 2s ] Uploading 'cdb-worker_1629282692_rutuja.dhimate.tar.gz' to 'cdb-stack-resource-backups-dev' succeeded.                   
[ - ] [ 0m 8s ] Rotating ASG instances...                                                                                                
[135, 144] in /Users/rutuja.dhimate/acquia-cloud/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-28303606f170/lib/plugins/rotate_asg_instances/asg.rb
   135:       def detach_instance(instance)
   136:         @step.success("Detaching instance: #{instance.instance_id}")
   137:         require 'byebug'; byebug
   138:         # If the ASG can't be brought up to capacity, re-attach the instance.
   139:         begin
=> 140:           instance.detach(should_decrement_desired_capacity: false)
   141:           @step.success('- Waiting for the AutoScaling '\
   142:                        'Group to be up to capacity')
   143:           wait_for_capacity
   144:         rescue Aws::AutoScaling::Errors::ValidationError => e
(byebug) instance.instance_id
"i-0399d88e625d16e7e"
(byebug) instance_id
[ ✓ ] [ 1m 4s ] Detaching instance: i-0399d88e625d16e7e*** NameError Exception: undefined local variable or method `instance_id' for #<Moonshot::RotateAsgInstances::ASG:0x00007fa3ef41c9c8>
Did you mean?  instance
               instance_of?

nil
(byebug) instance.id
"i-0399d88e625d16e7e"
(byebug) continue
[ | ] [ 8m 50s ] Shutting down i-0399d88e625d16e7e                                                                                       
[135, 144] in /Users/rutuja.dhimate/acquia-cloud/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-28303606f170/lib/plugins/rotate_asg_instances/asg.rb
   135:       def detach_instance(instance)
   136:         @step.success("Detaching instance: #{instance.instance_id}")
   137:         require 'byebug'; byebug
   138:         # If the ASG can't be brought up to capacity, re-attach the instance.
   139:         begin
=> 140:           instance.detach(should_decrement_desired_capacity: false)
   141:           @step.success('- Waiting for the AutoScaling '\
   142:                        'Group to be up to capacity')
   143:           wait_for_capacity
   144:         rescue Aws::AutoScaling::Errors::ValidationError => e
(byebug) continue
[ ✓ ] [ 15m 54s ] ASG instances rotated successfully!                                                                                    
[ ✓ ] [ 0m 6s ] Outdated instances removed successfully!