Investigate the redis High CPU and Memory Alarms

[ben851]

Description

As a developer/operator of GC Notify, I would like to only be alerted when there are actual issues with our system, and not during false alarms so that I do not get alert fatigue and am able to quickly identify real errors.

This card covers the following alerts in the alarm review spreadsheet

• redis-elasticache-high-cpu-warning-CacheCluster001CPUUtilization
• redis-elasticache-high-cpu-warning-CacheCluster002CPUUtilization
• redis-elasticache-high-cpu-warning-CacheCluster003CPUUtilization
• redis-elasticache-high-db-memory-critical-CacheCluster003
• redis-elasticache-medium-cpu-warning-CacheCluster001CPUUtilization
• redis-elasticache-medium-cpu-warning-CacheCluster002CPUUtilization

WHY are we building?

We are receiving a lot of noise in our operations slack channel that are not indicative of actual issues.

WHAT are we building?

Investigate the redis alarms and determine if they can be fixed or if the alarm needs adjustment

VALUE created by our solution

Fewer false alarms will increase developer agility and response times to actual issues.

Acceptance Criteria

• [ ] Redis alarms are clearly understood and documented
• [ ] Redis alarm is adjusted to reduce noise

QA Steps

• [ ] Verify that Redis alarms are not being spammed in staging