Near No-Downtime: Redis Backup and Restoration Strategy

[LindseySaari]

9/10/2024: This ticket was determined to be out of scope for the Redis POAM (a _nice to have rather than a must have, so it's being rescheduled and placed in the Near No-Downtime Upgrades Epic.

Description

Currently with the Redis tool, there is only a once-daily backup. There are replicas that are captured (as read-only) on a regular basis throughout the day that can be promoted, but it can not be edited. In order to have a more solid, reliable and consistent backup, in the event of a system failure, it is necessary to have a more regularly cadenced backup for Redis. Redis only offers a once-daily backup currently and it won't matter to VA if we have a 24-hour backup or a 12-hour backup; if data is lost it with the cache that will not be a good thing. The Rails cache, Apps cache and sidekiq Redis instances will need to be reviewed.

Tasks

• [ ] Review the Rails cache in order to understand the current state
• [ ] Review the Apps cache in order to understand the current state
• [ ] Review the Sidekiq Redis instances in order to understand the current state
• [ ] Investigate other ideas for data cache backup
• [ ] Document other ideas for data cache backup

Success Metrics

• [ ] There will be a solid, documented strategy for backing up and restoring the three Redis instances (Rails cache, Apps cache and sidekiq Redis instances), should a failure ensue.

Acceptability Criteria

• [ ] Discovery documentation for data cache backup from Redis that includes Rails cache, Apps cache and sidekiq Redis instances

[LindseySaari]

Notes: In terms of the issue were facing here, it doesn't appear that we would game anything from partitioning the data. ElastiCache for Redis Cluster Mode introduces sharding and automatic data partitioning across multiple nodes. If our primary concern is preventing data loss in the event of a Redis cluster failure, focusing on data durability and disaster recovery strategies seems to be key here. While Redis Cluster Mode enhances availability, scalability, and performance through sharding and automatic failover, we could benefit by complementing these features with a more robust backup and recovery process.

• Backup during the day
• Backup should be something different - should be triggered based on if we are in crisis. If there is an error scenario,
• We may want to explore persisting to Postgres or something - Postgres may provide a more robust approach to data persistance. By persisting critical job data to Postgres, we could ensure that even if Redis data is lost due to a failure, we would have a record of the jobs and their data that were queued.
• Review any benefits of cluster mode?
• Research monitoring and alerting for Redis. Monitoring metrics can help detect and respond to issues before they lead to data loss

[LindseySaari]

Postgres approach notes:

We could potentially utilize Sidekiq's middleware or lifecycle hooks to interact with Postgres. We could for example:

• Insert a record into Postgres when a job is enqueued.
• Update the record when the job starts, finishes, or fails.
• Store job results or error messages upon completion or failure.

This redundancy will act as a layer of protection to prevent data loss in the event of a disaster. We will want to ensure that the additional database calls do not negatively impact performance.