Upgrade AWS Elasticache Redis to the 5.0.x engine

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:

https://depo-platform-documentation.scrollhelp.site/index.html

282 stars 203 forks source link

Upgrade AWS Elasticache Redis to the 5.0.x engine #7545

Closed johnpaulashenfelter closed 4 years ago

johnpaulashenfelter commented 4 years ago

Our Redis instances in production/etc are running v3.2.4 which is really old. Newer versions of sidekiq require at least Redis 4.0 and newer versions of Rails need a higher version of the redis gem so we have a dependency train:

redis gem 4.0 or higher (needed for using the Rails RedisCache store) -> requires sidekiq 5.0 or higher (sidekiq 4.x is locked to redis gem ~ 3.2) -> requires redis server 4.0 or higher

We need to upgrade our ElasticCache redis engine to at least 4.0, preferably 5.0

[x] dsva-vetsgov-prod
[x] dsva-vetsgov-stag
[x] dsva-vetsgov-sandbox
[x] dsva-vetsgov-dev

johnpaulashenfelter commented 4 years ago

More info here: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/VersionManagement.html

johnpaulashenfelter commented 4 years ago

@wyattwalter @nathanhruby do you feel like we can just upgrade it in place? or should we practice that on another redis we spin up? Spinning up a new one would practice the recovery process as well if that's a process we even have.

What's the right way to plan this out? Assign an ops person to pair with me on it? /cc @ricetj

nathanhruby commented 4 years ago

Hey JP, I'm not sure what the nest way forward is here. Probably an upgrade in place would work, but I don't know how durable the data is during that (IIRC, aws's answer is "it varies"). Do you want to chat about on Tu/Wed real quick ?

omgitsbillryan commented 4 years ago

Looks like we're already at 5?

Source: https://console.amazonaws-us-gov.com/elasticache/home?region=us-gov-west-1#redis:

omgitsbillryan commented 4 years ago

Oh, duh - those vagov cluster groups are not in use by vets-api which is currently configured to use the vetsgov flavor instead. https://github.com/department-of-veterans-affairs/devops/blob/master/ansible/deployment/config/vets-api/prod-settings.local.yml.j2#L138

omgitsbillryan commented 4 years ago

Some notes:

vets-api staging is already using a redis in vagov
- prod & dev are still using redis in vetsgov
a redis cluser group already exists for prod & dev but vets-api isn't using it yet
sidekiq & vets-api redis data (i.e. redis_store.rb, cache_aside.rb) both use a singular redis instance
- would be a Nice-to-have if these could be two separate redis instances

erluti commented 4 years ago

PR for Sidekiq to be upgraded to 5 https://github.com/department-of-veterans-affairs/vets-api/pull/4128

johnpaulashenfelter commented 4 years ago

Downtime in #8184