Propose to ITRB that we use rolling Serverless ElastiCache Redis

[gaurav]

The idea is that our process for future NodeNorm upgrades should be:

1. [RENCI] Upload NodeNorm Redis RDB files to S3
2. [ITRB] Create new Serverless Redis RDB servers by restoring them from the S3 RDB files
3. [RENCI] Test Serverless servers, confirm that they are correctly loaded
4. [RENCI] Update Helm chart for nn-web with hostnames and passwords for the new Serverless Redis RDB servers
5. [ITRB] Redeploy nn-web with the new servers.
   • IF deployment fails: deploy previous version of the code and wait for updates
     6. [ITRB] Delete old Serverless Redis RDB servers
     7. [ITRB] Make database backups of the Serverless Redis RDB servers

We need to:

• Check with @YaphetKG to make sure this makes sense
• Discuss if this is feasible with ITRB
• Do a test deployment of this on CI
• If that works, replace Test and Prod with this system

[gaurav]

A back-of-the-envelope calculation suggests that this will be more expensive than the per-node we're currently playing for. A better option for us is Data tiering, which offloads some of the database onto an SSD, which we have tried successfully on ITRB. Closing this as something we want to try to investigate.