NOD | V2 Launch Monitor

[saderagsdale]

How to use this ticket

This is a daily checklist for monitoring the health of our release. Tasks for this ticket should be completed first thing at the beginning of each day, and reported to the internal team and key stakeholders. Below are the metrics used to gauge the health of the release. See these links for details on the release and rollback plan, and incident monitoring plan.

Step 1: Check the monitoring dashboards for release health

• [ ] Engineering: Post the FE and BE error metrics for the previous day, including the type of error and intervention needed.
• [ ] Product: Check Medallia feedback by submitting a qualitative data request

Dashboards

NOD Dashboard in DataDog Launch Monitoring (V1 v. V2). NOD Domo Dashboard

Are users abandoning the form disproportionately between V1 and V2?

• Form completion - filter by funnel version

Is the error rate showing any spikes?

• NOD form submission error alert
• NOD/SC evidence submission error alert.

Is the contestable issues endpoint working?

• NOD - Get Contestable Issues API traffic anomaly check
• NOD - Get Contestable Issues - Error rate

Is submission traffic proportionate to the typical submission rate on a given day?

• NOD - Overall Claim Submission traffic is low
• NOD - Overall Claim Submission API traffic anomaly check
• NOD - Overall Claim Submission - Errors
• Traffic on NOD submission endpoint greater than or equal to 12 submissions / day.

Is evidence submission proportionate to the typical submission rate on a given day?

• NOD/SC - Evidence Upload to Lighthouse - Permanent Errors
• NOD and SC Evidence traffic is low
• NOD and SC Evidence traffic anomaly check
• Traffic on NOD/SC evidence submission endpoint greater than or equal to 100 uploads / day.

Do our evidence uploads match Lighthouse's count?

• NOD - Evidence Uploads to Lighthouse API traffic anomaly check
• NOD - Evidence Upload to S3 API traffic anomaly check

Alerts will be automatically posted in the #benefits-decision-reviews-notifications Slack channel, and sent to team members via email. Eugene and Sade will be responsible for triaging these alerts daily.

Step 2: Document new bugs or spikes using this template and list them below.

Step 3: Update the enablement team and LH Banana Peels team

• [ ] Product: Craft an update with input from engineering, share with PO and LH Banana Peels team.
• [ ] Product: Check for opportunities to expedite the release.
• [ ] Product: Communicate any changes in the release plan to the stakeholders above and update the release plan.

Step 4: Confirm when release period is officially closed

• [ ] Product: At the end of the observation period, get buy-in from PO and LH Banana Peels team before increasing the release.

[saderagsdale]

• Robin and Rocio will conduct the canary. @Mottie
• 25% increments, a few days apart

[saderagsdale]

Need to update logging links, KPIs @saderagsdale @data-doge

[saderagsdale]

Needs KPIs for NOD production validation. Devs will chat tomorrow.

[data-doge]

Note: The changes in V2 will only affect the submit action, which we are already monitoring and sending alerts for in our NOD dashboard. So we are all good to go for this ticket, once launched.

[saderagsdale]

@data-doge can you tag me when the KPIs we're monitoring for the release are listed and linked in the ticket?

[saderagsdale]

@data-doge just waiting for updated KPIs for this one.

[data-doge]

Sorry for the delay on this @saderagsdale - just updated KPIs in the PR description.

[saderagsdale]

Need to update the analytics for this. @saderagsdale will reach out to platform

[HeatherWidmont]

@data-doge to send another batch of UUIDs to Tim

[data-doge]

I'll send that batch of UUIDs to Tim tomorrow - ran out of time today

[saderagsdale]

Sade will add links from release plan draft.

[saderagsdale]

Eugene and Sade will update with new monitor links (as needed).

[saderagsdale]

Need to add S3 auditing for parity.

[data-doge]

@saderagsdale Made just one update to ticket desc:

Traffic on NOD/SC evidence submission endpoint greater than or equal to ~150~ 100 uploads / day. Tracked via this DataDog alert.

Our monitor was too sensitive and was giving too many false positives, so I changed the evidence submission threshold to 100 / day.

[HeatherWidmont]

Dialed back down to 0% due to an issue on the LH side, may dial back up today. Just tested a fix from LH so we'll see what they say

[HeatherWidmont]

Back up to 25% as of yesterday afternoon, monitors look pretty good so far

[anniebtran]

Julie gave us the OK to increase to 50% 🎉

[anniebtran]

Increased to 50% 👍

[HeatherWidmont]

At 100%, things are looking good