Implement Actionable Alerts System

[RachalCassity]

Summary

As our system grows in complexity, it's becoming increasingly crucial to implement a robust actionable alerts system. Currently, our alerting mechanism notifies us of potential issues, but lacks the capability to provide actionable insights or automated responses. This ticket aims to address this gap by developing a comprehensive actionable alerts system.

Tasks

• [ ] Create an alert template to standardization alerting and monitoring.
• [ ] Include recommended steps or actions to mitigate the problem.
• [ ] Includes Datadog Logs, Error Tracking, APM metrics, if applicable.
• [ ] Provide insight as to why the alerted value was chosen. Example: This {{DD ALERT}} was changed from 95% to 40% because production broke when {{DD ALERT}} was around 40%.

Let's get our alerts system to the next level with actionable insights and automated responses! 🚀🔔

[jennb33]

7/15 update from refinement meeting: @rmtolmach noted that when she is addressing an alert, she makes the changes and updates in real time but a comprehensive audit of all monitors may be necessary. Need more information for templating (just Platform? or VFS teams as well) and to review this with Rachal when she is back in office.