2024 Q1 PACT Act Support: External Service Monitor and Discovery

[jennb33]

Summary

As we prepare for the third round of PACT Act submissions, our objective is to ensure robust monitoring of all external services that are impacted. We aim to prevent overloading and avoid the errors encountered in the first round of submissions.

Goals

• [x] Identify all external systems affected by PACT Act submissions.
• [x] Ensure proper configuration of circuit breakers and corresponding alerts.
• [x] Update and flag the breakers chart in DataDog for easy monitoring.

Background

In the previous rounds of submissions, we experienced critical issues due to overloading of downstream systems (by submitting too large of uploads). By setting up and optimizing external service monitors, we can proactively detect and respond to such issues, minimizing system downtime and ensuring compliance with the PACT Act requirements.

Action Items

• [x] Review the interaction flow of submissions to map out all external systems involved.
• [x] Audit current monitoring setup to identify any gaps in the coverage.
• [x] Set up additional monitors and alerts for newly identified external services.
• [x] Configure DataDog breakers chart to flag potential overloads or errors.
• [x] Validate that circuit breaker alerts are functioning correctly and are appropriately calibrated.
• [x] Conduct a test to ensure that monitoring and alerts respond as expected.

Importance of Monitoring

Monitoring is crucial as it provides real-time visibility into system health, performance, and reliability. It allows us to:

• Detect and respond to issues before they affect users.
• Understand the impact of the surge in submissions on external services.
• Maintain service availability by preventing cascading failures.
• Ensure that our system's resilience mechanisms are effectively engaged.

By being proactive with our monitoring strategy, we safeguard the integrity of our services and uphold the trust of our users.

Expected Outcomes

• A complete set of monitors for all external services related to PACT Act submissions.
• Enhanced visibility into system health via DataDog dashboards.
• Reliable alerting mechanisms that trigger before system thresholds are breached.
• Improved system resilience against high load and potential downstream errors.

[jennb33]

Next steps: find the breakers dashboard and link to it in the BE Support documentation

[rmtolmach]

Next steps: find the breakers dashboard and link to it in the BE Support documentation

Technically done, but I want to add more to the docs, need help from @LindseySaari : https://vfs.atlassian.net/wiki/spaces/PTST/pages/2183037084/Backend+Support+Responsibilities#Breakers-dashboards

[rmtolmach]

Updated the doc. Done.