search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
282 stars 203 forks source link

Scheduled downtime for Elasticache production upgrade #8184

Closed johnpaulashenfelter closed 4 years ago

johnpaulashenfelter commented 4 years ago

Following a discussion on the va-eng-leads Slack channel and this document: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/platform/engineering/scheduled-downtime-process.md

Proposed downtime for vets-api

Systems affected: vets-api (effectively nearly all services, since redis stores authentication, caches other data) Proposed date and time for staging: N/A -- staging is already running Redis v5 Proposed date and time for production: Saturday 4/25/2020 from 9:00 PM to 9:02 PM (approximately)

Time is honestly anytime in our Saturday "preapproved" window, but assuming we'll aim for around 9PM Eastern.

Additional steps:

drorva commented 4 years ago

Approved.

johnpaulashenfelter commented 4 years ago

The various folks on PagerDuty schedules for this time period:

Also mentioned to them in Slack. This is mostly an "inform" step unless @nathanhruby is assigned the execution.

johnpaulashenfelter commented 4 years ago

Draft plan:

Redis migration

Sign into AWS Console

Amazon Web Services Sign-In

Scale worker ASG to zero

https://console.amazonaws-us-gov.com/ec2/autoscaling/home?region=us-gov-west-1#AutoScalingGroups:view=details

Choose worker Choose Edit Desired capacity and min to 0 Save (Wait)

Backup redis

https://console.amazonaws-us-gov.com/elasticache/home?region=us-gov-west-1#redis:

Choose dsva-vetsgov-prod Choose Actions > Backup (Wait)

Upgrade redis engine

Still on page (https://console.amazonaws-us-gov.com/elasticache/home?region=us-gov-west-1#redis:)

Choose Actions > Modify Choose Engine Version > 5.0.6 (Wait

Scale worker ASG back up

https://console.amazonaws-us-gov.com/ec2/autoscaling/home?region=us-gov-west-1#AutoScalingGroups:view=details

Choose worker Choose Edit Desired capacity and min to 4 Save (Wait)

johnpaulashenfelter commented 4 years ago

Current environments

Prod redis: “dsva-vetsgov-prod.eqe4uk.ng.0001.usgw1.cache.amazonaws.com” version: 3.2.4 vpc: old

Dev redis: “dsva-vetsgov-dev.eqe4uk.ng.0001.usgw1.cache.amazonaws.com” version: 5.0.6 vpc: old

Sandbox redis: “dsva-vagov sandbox.eqe4uk.ng.0001.usgw1.cache.amazonaws.com” version: 5.0.6 vpc: VAEC

Staging redis: “dsva-vagov-staging.eqe4uk.ng.0001.usgw1.cache.amazonaws.com” version: 5.0.5 vpc: VAEC

Future migration

Unused

johnpaulashenfelter commented 4 years ago

Successful migration completed as expected.