Closed tanujjain49 closed 2 years ago
MANUAL REVIEW
2022-01-06 12:59:45 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$ git diff
diff --git a/Gemfile b/Gemfile
index 399baee..68fc232 100644
--- a/Gemfile
+++ b/Gemfile
@@ -52,8 +52,8 @@ group :development do
# @see https://github.com/acquia/moonshot/pull/245
# @see https://backlog.acquia.com/browse/CPD-3865
gem 'moonshot',
- git: 'git@github.com:acquia/moonshot.git',
- ref: 'proxy-cli-hook-aws-v2'
+ git: 'git@github.com:tanujjain49/moonshot.git',
+ ref: 'CPD-6858'
gem 'moonshot-production-safety',
git: 'git@github.com:acquia/cloud-moonshot-production-safety.git',
diff --git a/Gemfile.lock b/Gemfile.lock
index d178d59..b584d34 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -99,9 +99,23 @@ GIT
fpm (> 1.4)
GIT
- remote: git@github.com:acquia/moonshot.git
- revision: e7e923ca13440303cad6951944ead763bf2c54a7
- ref: proxy-cli-hook-aws-v2
+ remote: git@github.com:acquia/signalfx-bugsnag-middleware.git
+ revision: a5f1c13593888caf7002bfb011b3213057031ffb
+ ref: master
+ specs:
+ signalfx-bugsnag-middleware (0.1.0)
+
+GIT
+ remote: git@github.com:acquia/systemd-daemon.git
+ revision: 44623468b56dcd945242e313bef1cd6aeb48068b
+ ref: 0.0.1
+ specs:
+ systemd-daemon (0.1.0)
+
+GIT
+ remote: git@github.com:tanujjain49/moonshot.git
+ revision: bfd7dba55044f0d51ea0986539da4a1cd2da99f6
+ ref: CPD-6858
specs:
moonshot (2.0.0.beta6)
activesupport
@@ -119,20 +133,6 @@ GIT
travis
vandamme
-GIT
- remote: git@github.com:acquia/signalfx-bugsnag-middleware.git
- revision: a5f1c13593888caf7002bfb011b3213057031ffb
- ref: master
- specs:
- signalfx-bugsnag-middleware (0.1.0)
-
-GIT
- remote: git@github.com:acquia/systemd-daemon.git
2022-01-06 13:00:04 [cloudservicesdev|cloud-data:tanujj
dev-tanuj
stage.
2022-01-06 13:02:09 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$ bundle exec moonshot update -n dev-tanuj
Using the Central Model for authentication
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.
[ ✓ ] [ 0m 8s ] ChangeSet moonshot-cdb-worker-dev-tanuj-1641454378 ready!
* Modify LaunchConfig (AWS::AutoScaling::LaunchConfiguration)
- Will be replaced
- Caused by template change (Properties: ImageId)
* Modify WorkerASG (AWS::AutoScaling::AutoScalingGroup)
- May be replaced (Conditional)
- Caused by LaunchConfig (ResourceReference)
Apply changes? yes
[ ✓ ] [ 0m 2s ] Executed ChangeSet moonshot-cdb-worker-dev-tanuj-1641454378 for CloudFormation Stack cdb-worker-dev-tanuj.
[ ✓ ] [ 0m 30s ] CloudFormation Stack cdb-worker-dev-tanuj successfully updated.
[ ✓ ] [ 0m 3s ] CodeDeploy Application cdb-worker-dev-tanuj already exists.
[ ✓ ] [ 0m 0s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-tanuj already exists.
[ ✓ ] [ 0m 6s ] AutoScaling Group(s) up to capacity!
[ ✓ ] [ 0m 2s ] Uploading 'cdb-worker_1641454547_tanuj.jain.tar.gz' to 'cdb-stack-resource-backups-dev' succeeded.
--> ASG desired capacity updated to 2.
--> All instances cycled.
--> ASG desired capacity updated to 1.
[ ✓ ] [ 9m 3s ] ASG instances rotated successfully!
2022-01-06 13:14:55 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$
MANUAL REVIEW
2022-01-11 15:17:11 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$ bundle exec moonshot update -n dev-tanuj
Using the Central Model for authentication
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
[ ✓ ] [ 0m 1s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 0s ] Using previous encrypted value for NewRelicLicenseKey.
[ ✓ ] [ 0m 15s ] ChangeSet moonshot-cdb-worker-dev-tanuj-1641894465 ready!
* Modify LaunchConfig (AWS::AutoScaling::LaunchConfiguration)
- Will be replaced
- Caused by template change (Properties: ImageId)
* Modify WorkerASG (AWS::AutoScaling::AutoScalingGroup)
- May be replaced (Conditional)
- Caused by LaunchConfig (ResourceReference)
Apply changes? y
Please enter 'yes' or 'no'!
Apply changes? yes
[ ✓ ] [ 0m 3s ] Executed ChangeSet moonshot-cdb-worker-dev-tanuj-1641894465 for CloudFormation Stack cdb-worker-dev-tanuj.
[ ✓ ] [ 0m 28s ] CloudFormation Stack cdb-worker-dev-tanuj successfully updated.
[ ✓ ] [ 0m 2s ] CodeDeploy Application cdb-worker-dev-tanuj already exists.
[ ✓ ] [ 0m 0s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-tanuj already exists.
[ ✓ ] [ 0m 5s ] AutoScaling Group(s) up to capacity!
[ ✓ ] [ 0m 2s ] Uploading 'cdb-worker_1641894539_tanuj.jain.tar.gz' to 'cdb-stack-resource-backups-dev' succeeded.
--> ASG desired capacity updated to 2.
[ ✓ ] [ 3m 54s ] AutoScaling Group up to capacity!
[295, 304] in /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-e7e923ca1344/lib/plugins/rotate_asg_instances/asg.rb
295: instance.wait_until_stopped
296: end
297:
298: def instance_in_terminal_state?(instance)
299: require 'byebug'; byebug
=> 300: state = if instance.is_a?(Aws::EC2::Instance)
301: instance.state.name
302: elsif instance.is_a?(Aws::AutoScaling::Instance)
303: instance.load.lifecycle_state
304: end
(byebug) instance.id
"i-022f79d23bbb2119a"
(byebug) instance.is_a?(Aws::EC2::Instance)
false
(byebug) continue
--> ASG desired capacity updated to 1.
[ / ] [ 9m 33s ] Waiting for Autoscaling group to reach desired capacity...#<Thread:0x00007fd424028bb8@/Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/gems/interactive-logger-0.1.3/lib/interactive-logger.rb:39 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
8: from /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/gems/interactive-logger-0.1.3/lib/interactive-logger.rb:40:in `block in start_threaded'
7: from /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-e7e923ca1344/lib/plugins/rotate_asg_instances/asg.rb:42:in `block in rotate_asg_instances'
6: from /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-e7e923ca1344/lib/plugins/rotate_asg_instances/asg.rb:69:in `with_scale_up'
5: from /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-e7e923ca1344/lib/plugins/rotate_asg_instances/asg.rb:43:in `block (2 levels) in rotate_asg_instances'
4: from /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-e7e923ca1344/lib/plugins/rotate_asg_instances/asg.rb:103:in `outdated_volumes'
3: from /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-e7e923ca1344/lib/plugins/rotate_asg_instances/asg.rb:103:in `each'
2: from /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-e7e923ca1344/lib/plugins/rotate_asg_instances/asg.rb:105:in `block in outdated_volumes'
1: from /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-e7e923ca1344/lib/plugins/rotate_asg_instances/asg.rb:303:in `instance_in_terminal_state?'
/Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/aws-sdk-ruby-01ea5bf78a70/aws-sdk-resources/lib/aws-sdk-resources/resource.rb:223:in `block in add_data_attribute': undefined method `[]' for nil:NilClass (NoMethodError)
[ ✗ ] [ 9m 33s ] Error while performing step: Rotating ASG instances...
NoMethodError: undefined method `[]' for nil:NilClass
undefined method `[]' for nil:NilClass (at /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/aws-sdk-ruby-01ea5bf78a70/aws-sdk-resources/lib/aws-sdk-resources/resource.rb:223:in `block in add_data_attribute')
2022-01-11 15:28:37 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$
-> Instance state in aws console is terminated before we continued in byebug session.
Updated the moonshot gem to use the fixed changes and ran the moonshot update on another stage. ( dev-pritam ).
2022-01-11 15:30:41 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$ git diff
diff --git a/Gemfile b/Gemfile
index 399baee..68fc232 100644
--- a/Gemfile
+++ b/Gemfile
@@ -52,8 +52,8 @@ group :development do
# @see https://github.com/acquia/moonshot/pull/245
# @see https://backlog.acquia.com/browse/CPD-3865
gem 'moonshot',
- git: 'git@github.com:acquia/moonshot.git',
- ref: 'proxy-cli-hook-aws-v2'
+ git: 'git@github.com:tanujjain49/moonshot.git',
+ ref: 'CPD-6858'
gem 'moonshot-production-safety',
git: 'git@github.com:acquia/cloud-moonshot-production-safety.git',
diff --git a/Gemfile.lock b/Gemfile.lock
index b01c5dd..f292860 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -99,9 +99,23 @@ GIT
fpm (> 1.4)
GIT
- remote: git@github.com:acquia/moonshot.git
- revision: e7e923ca13440303cad6951944ead763bf2c54a7
- ref: proxy-cli-hook-aws-v2
+ remote: git@github.com:acquia/signalfx-bugsnag-middleware.git
+ revision: a5f1c13593888caf7002bfb011b3213057031ffb
+ ref: master
+ specs:
+ signalfx-bugsnag-middleware (0.1.0)
+
+GIT
+ remote: git@github.com:acquia/systemd-daemon.git
+ revision: 44623468b56dcd945242e313bef1cd6aeb48068b
+ ref: 0.0.1
+ specs:
+ systemd-daemon (0.1.0)
+
+GIT
+ remote: git@github.com:tanujjain49/moonshot.git
+ revision: bfd7dba55044f0d51ea0986539da4a1cd2da99f6
+ ref: CPD-6858
specs:
moonshot (2.0.0.beta6)
activesupport
@@ -119,20 +133,6 @@ GIT
travis
vandamme
-GIT
-GIT
PATH
remote: .
specs:
2022-01-11 15:31:06 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$
2022-01-11 15:31:07 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$
2022-01-11 15:31:07 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$ bundle exec moonshot update -n dev-tanuj
Using the Central Model for authentication
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.
SSH failed. exit status: 255 output: Pseudo-terminal will not be allocated because stdin is not a terminal.
ssh: Could not resolve hostname /bin/true: nodename nor servname provided, or not known error: Pseudo-terminal will not be allocated because stdin is not a terminal.
ssh: Could not resolve hostname /bin/true: nodename nor servname provided, or not known (at /Users/tanuj.jain/Documents/github/acquia/cloud-database-worker/vendor/bundle/ruby/2.6.0/bundler/gems/moonshot-bfd7dba55044/lib/plugins/rotate_asg_instances/ssh.rb:16:in `block in test_ssh_connection')
2022-01-11 15:32:34 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$ bundle exec moonshot update -n dev-tanuj
Using the Central Model for authentication
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.
[ ✗ ] [ 0m 1s ] ChangeSet failed to create: The submitted information didn't contain changes. Submit different information to create a change set.
[ ✓ ] [ 0m 2s ] CodeDeploy Application cdb-worker-dev-tanuj already exists.
[ ✓ ] [ 0m 0s ] CodeDeploy CodeDeploy Deployment Group cdb-worker-dev-tanuj already exists.
[ ✓ ] [ 0m 4s ] AutoScaling Group(s) up to capacity!
[ ✓ ] [ 0m 2s ] Uploading 'cdb-worker_1641895450_tanuj.jain.tar.gz' to 'cdb-stack-resource-backups-dev' succeeded.
[ ✓ ] [ 0m 3s ] No outdated instances to rotate.
2022-01-11 15:34:18 [cloudservicesdev|cloud-data:tanujjain] ~/Documents/github/acquia/cloud-database-worker$ bundle exec moonshot update -n dev-pritam
Using the Central Model for authentication
Version 2 of the Ruby SDK will enter maintenance mode as of November 20, 2020. To continue receiving service updates and new features, please upgrade to Version 3. More information can be found here: https://aws.amazon.com/blogs/developer/deprecation-schedule-for-aws-sdk-for-ruby-v2/
[ ✓ ] [ 0m 0s ] Using existing KMS Key for ParameterKMS!
[ ✓ ] [ 0m 1s ] Using previous encrypted value for NewRelicLicenseKey.
[ ✓ ] [ 0m 15s ] ChangeSet moonshot-cdb-worker-dev-pritam-1641895495 ready!
-> instance state was terminated before we continued in the byebug session.
Manual test looks good, just needs some unit tests I think.
Updated the unit tests and have refactored the code for instance_terminal_state?.
I want to see a manual review or a test covering these changes. Preferably reproducing the terminated instance issue.
Hint: we can just call moonshot update after making a change to worker template(eg: update ami) and that will trigger rotation of instances after stack update.