CPD-6267: Updated asg plugin to scale up before rotation.

kaushik commented 3 years ago

Manual Review Steps: When Desired capacity < Max capacity on ASG

Check the desired capacity of the asg.
Make a change to launch config(eg: update ami id) and perform moonshot push
Observe that a new instance is added to the ASG followed by rotation.
confirm that the desired capacity is back to original value.

When Desired capacity = Max capacity on ASG

Check the desired capacity of the asg.
Make a change to launch config(eg: update ami id) and perform moonshot push
Observe that an existing instance is detached from the ASG followed by rotation and the number of instances is at original desired capacity from previous step.
confirm that the desired capacity is same as original value.

kaushik commented 3 years ago

├─ ASG: WorkerASG
│
│ Name: cdb-worker-dev-kaushik-WorkerASG-J9A4TDUS23ZH
│ Using ELB health checks, with a 180s health check grace period.
│ Desired Capacity is 1 (Min: 1, Max: 3).
│ Has 1 Load Balancer(s): cdb-worker-dev-ELB-1PVNZN1DLI6CW

While waiting for rotation to complete we can see that there are two instances in the asg instead of 1 which is the desired capacity.

[cloudservicesdev|hosting-dev:kaushiksirineni] ~/devel/repos/cloud-database-worker$ be moonshot update -n dev-kaushik
.
[ \ ] [ 1m 59s ] Instances:  i-00098c6182062a51e (InService),  i-072116adf6a9fb803 (Pending:Wait)

Final output of the moonshot update

[cloudservicesdev|hosting-dev:kaushiksirineni] ~/devel/repos/cloud-database-worker$ be moonshot update -n dev-kaushik
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
SIGNALFX_API_TOKEN environment variable is empty, SignalFx reporting has been disabled.
[ ✓ ] [ 0m 1s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 0s ] Using previous encrypted value for NewRelicLicenseKey.
[ ✓ ] [ 0m 11s ] ChangeSet moonshot-cdb-worker-dev-kaushik-1634094589 ready!
* Modify LaunchConfig (AWS::AutoScaling::LaunchConfiguration)
 - Will be replaced
 - Caused by template change (Properties: ImageId)
* Modify WorkerASG (AWS::AutoScaling::AutoScalingGroup)
 - May be replaced (Conditional)
 - Caused by LaunchConfig (ResourceReference)
Apply changes? yes
[ ✓ ] [ 0m 1s ] Executed ChangeSet moonshot-cdb-worker-dev-kaushik-1634094589 for CloudFormation Stack cdb-worker-dev-kaushik.
[ ✓ ] [ 0m 32s ] CloudFormation Stack cdb-worker-dev-kaushik successfully updated.
[ ✓ ] [ 0m 1s ] CodeDeploy Application cdb-worker-dev-kaushik already exists.
[ ✓ ] [ 0m 0s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-kaushik already exists.
[ ✓ ] [ 0m 1s ] AutoScaling Group(s) up to capacity!
[ ✓ ] [ 0m 0s ] Uploading 'cdb-worker_1634094639_kaushik.sirineni.tar.gz' to 'cdb-stack-resource-backups-dev' succeeded.
--> ASG desired capacity updated to 2.
--> All instances cycled.
--> ASG desired capacity updated to 1.
[ ✓ ] [ 9m 7s ] ASG instances rotated successfully!

Desired capacity was set back to 1 moonshot status

├─ ASG: WorkerASG
│
│ Name: cdb-worker-dev-kaushik-WorkerASG-J9A4TDUS23ZH
│ Using ELB health checks, with a 180s health check grace period.
│ Desired Capacity is 1 (Min: 1, Max: 3).
│ Has 1 Load Balancer(s): cdb-worker-dev-ELB-1PVNZN1DLI6CW

kaushik commented 3 years ago

Manual review contd...

Updated the ASG to have same desired and max capacity

[cloudservicesdev|hosting-dev:kaushiksirineni] ~/devel/repos/cloud-database-worker$ be moonshot update -n dev-kaushik -P ASGMaxCapacity=2 -P ASGDesiredCapacity=2
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
SIGNALFX_API_TOKEN environment variable is empty, SignalFx reporting has been disabled.
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.
[ ✓ ] [ 0m 11s ] ChangeSet moonshot-cdb-worker-dev-kaushik-1634095340 ready!
* Modify WorkerASG (AWS::AutoScaling::AutoScalingGroup)
 - Caused by template change (Properties: MaxSize)
 - Caused by ASGDesiredCapacity (ParameterReference)
 - Caused by ASGMaxCapacity (ParameterReference)
 - Caused by template change (Properties: DesiredCapacity)
Apply changes? yes
[ ✓ ] [ 0m 1s ] Executed ChangeSet moonshot-cdb-worker-dev-kaushik-1634095340 for CloudFormation Stack cdb-worker-dev-kaushik.
[ ✓ ] [ 0m 21s ] CloudFormation Stack cdb-worker-dev-kaushik successfully updated.
[ ✓ ] [ 0m 1s ] CodeDeploy Application cdb-worker-dev-kaushik already exists.
[ ✓ ] [ 0m 0s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-kaushik already exists.
[ ✓ ] [ 2m 8s ] AutoScaling Group(s) up to capacity!
[ ✓ ] [ 0m 1s ] Uploading 'cdb-worker_1634095506_kaushik.sirineni.tar.gz' to 'cdb-stack-resource-backups-dev' succeeded.
[ ✓ ] [ 0m 0s ] No outdated instances to rotate.

[cloudservicesdev|hosting-dev:kaushiksirineni] ~/devel/repos/cloud-database-worker$ be moonshot status -n dev-kaushik
.
├─ ASG: WorkerASG
│
│ Name: cdb-worker-dev-kaushik-WorkerASG-J9A4TDUS23ZH
│ Using ELB health checks, with a 180s health check grace period.
│ Desired Capacity is 2 (Min: 1, Max: 2).
│ Has 1 Load Balancer(s): cdb-worker-dev-ELB-1PVNZN1DLI6CW.
.

Updated the launch config and ran update... While in progress we only see two instances and one of them was being rotated.

[ | ] [ 8m 21s ] Instances:  i-021086dca5f382557 (InService),  i-0e160ccef7c5c733f (InService)

Final output of update shows that it did not do scaling before and after rotation

[cloudservicesdev|hosting-dev:kaushiksirineni] ~/devel/repos/cloud-database-worker$ be moonshot update -n dev-kaushik
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
SIGNALFX_API_TOKEN environment variable is empty, SignalFx reporting has been disabled.
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.
[ ✓ ] [ 0m 11s ] ChangeSet moonshot-cdb-worker-dev-kaushik-1634095592 ready!
* Modify LaunchConfig (AWS::AutoScaling::LaunchConfiguration)
 - Will be replaced
 - Caused by template change (Properties: ImageId)
* Modify WorkerASG (AWS::AutoScaling::AutoScalingGroup)
 - May be replaced (Conditional)
 - Caused by LaunchConfig (ResourceReference)
Apply changes? yes
[ ✓ ] [ 0m 1s ] Executed ChangeSet moonshot-cdb-worker-dev-kaushik-1634095592 for CloudFormation Stack cdb-worker-dev-kaushik.
[ ✓ ] [ 0m 23s ] CloudFormation Stack cdb-worker-dev-kaushik successfully updated.
[ ✓ ] [ 0m 0s ] CodeDeploy Application cdb-worker-dev-kaushik already exists.
[ ✓ ] [ 0m 1s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-kaushik already exists.
[ ✓ ] [ 0m 1s ] AutoScaling Group(s) up to capacity!
[ ✓ ] [ 0m 0s ] Uploading 'cdb-worker_1634095678_kaushik.sirineni.tar.gz' to 'cdb-stack-resource-backups-dev' succeeded.
--> All instances cycled.
[ ✓ ] [ 10m 39s ] ASG instances rotated successfully!

The desired capacity is unchanged

[cloudservicesdev|hosting-dev:kaushiksirineni] ~/devel/repos/cloud-database-worker$ be moonshot status -n dev-kaushik
├─ ASG: WorkerASG
│
│ Name: cdb-worker-dev-kaushik-WorkerASG-J9A4TDUS23ZH
│ Using ELB health checks, with a 180s health check grace period.
│ Desired Capacity is 2 (Min: 1, Max: 2).
│ Has 1 Load Balancer(s): cdb-worker-dev-ELB-1PVNZN1DLI6CW

acquia / moonshot

CPD-6267: Updated asg plugin to scale up before rotation. #277