cds-snc / notification-planning-core

Project planning for GC Notify Core Team

0 stars 0 forks source link

Prepare the monitoring for priority lanes outliers #385

Open jimleroyer opened 1 month ago

jimleroyer commented 1 month ago

What

Monitor the priority lanes for template outliers that are breaking the SLOs and should be moved into a different priority queue.

Why

As we are reworking how our sending vehicle and priority queues function, some of the templates will lose its current priority lane configuration and will fall back into the defaults of the predefined category template. It is possible that some of these default might not fit with the current SLOs profile and would need a customized priority lane configuration to be brought back to SLOs stability.

Acceptance Criteria

-[ ] Bring up the queries that are in the SLOs document into Blazer for live 1-day analysis. -[ ] Bring up the queries that are in the SLOs document into QuickSight for ad-hoc analysis (this routine chore). -[ ] Identify audit mechanism -- check if the template history table can save these changes so we might not have to maintain manually such a document

QA

[x] Review the query in Blazer and execute it with different parameters.
[ ] Review the query in Quicksight and execute it with different parameters.

Resources

Blazer and its queries
Auditing document

sastels commented 1 month ago

Started porting / writing queries in Blazer

sastels commented 1 month ago

This dashboard in Blazer shows templates that have taken longer than the 90% SLO to send (currently using the template's process_type).

grouped by days
puts together all the notifications for the particular template that have taken longer than the 90% SLO
shows the # of these slow notifications as well as the median time they took
only shows a template if many of them were slow on a given day

P0NDER0SA commented 1 month ago

Jimmy will take a look

sastels commented 1 month ago

:thinking: Do we want to exclude SMS sent through a dedicated number? These do not go through the low/medium/high priority queues.

sastels commented 1 month ago

We'll make a separate card for keeping an eye on dedicated numbers

jimleroyer commented 1 month ago

I checked the query and that looks good. 👍

sastels commented 1 month ago

Have the corresponding analysis mostly roughed in in QuickSight, just need the new Notifications with categories dataset.

sastels commented 3 weeks ago

Added template category priority lanes to the quicksight query and made a dashboard in the Notify shared folder, with the corresponding analysis in the Analyses WIP - make a copy for yourself instead of changing the original shared folder.