Respond to the discovery effort on Zero Silent Failures

[holdenhinkle]

Background

Matthew Dingee posted a request asking all VFS to review their products for silent failures. Here's the post - https://dsva.slack.com/archives/CE4304QPK/p1726600494318029

Respond to his request.

Tasks

• [x] Reviewed the How to discover if your application has a silent failure document
• [x] Respond accordingly

Acceptance Criteria

• [x] This has been handled

---

How to configure this issue

1. Label with practice area (backend, frontend,ux, design, research)
2. Label for MVP work? (mvp)
3. Add an estimate
4. Attach an Epic (what body of work is this a part of?)

[holdenhinkle]

I reviewed the How to discover if your application has a silent failure document and determined ARM's products do not have any silent failures.

ARM's products currently do not persist, process, or submit any forms or other requests like file uploads or performing an action (such as refilling a prescription or ordering supplies), etc..

I posted to the #zero-silent-failures Slack channel that we do not have any silent failures - https://dsva.slack.com/archives/C07KTTFP308/p1726771961196239

[holdenhinkle]

cc @parkerbees

[oddball-lindsay]

I was suspecting we were ok on the silent failures front and have been meaning to review the Managing Tagging Standards and guideline for Managing Errors to confirm.

Thanks @holdenhinkle for taking the lead and confirming!

[holdenhinkle]

Checklist

Start

• [x] Do you know when your application shipped to production? Yes, we know.
• [x] Did your application use the same APIs when it shipped as it does today? Yes

Monitoring

• [x] Do you monitor the API that you submit to via Datadog? Yes
• [ ] Does your Datadog monitoring use the appropriate tagging?
  • If not, implement tagging standards. Adding the dependency tag is highly recommended!
• [x] Do errors detected by Datadog go into a Slack notifications channel? Yes - #benefits-representation-management-notifications
• [x] Does more than one person look at the Slack notifications channel containing errors on a daily basis? Yes
• [x] Do the team members monitoring the Slack channel have a system for acknowledging and responding to the errors that appear there? Yes Find a Rep - https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/accredited-representation-management/product-documentation/find-a-representative/launch-materials/product-playbook-incident-response-plan.md Rep Status Widget - https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/accredited-representation-management/product-documentation/representative-status-widget/launch-materials/product-playbook-incident-response-plan.md

Reporting errors

• [x] Have you filed issues for errors that are appearing in Datadog / Slack? No. We haven't had any errors to file issues for.
• [x] Do all fatal errors thrown in your application end up visible to the end user either in the user interface or via email? Yes.

Documentation

• [ ] Do you have a diagram of the submission path that user data your application accepts takes to reach a system of record? Yes. Find a Rep - https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/accredited-representation-management/product-documentation/find-a-representative/launch-materials/arm-architecture-diagrams.pdf Rep Status Widget - https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/accredited-representation-management/product-documentation/representative-status-widget/launch-materials/architecture_diagrams.pdf
• [ ] Do you understand how the error is handled when each system in the submission path fails, is down for maintenance, or is completely down?
  • If not, then create documentation that captures how errors in each system are handled. Detail which systems retry a submission and what happens when those retries exhaust. Show this in your diagram.
• [x] Has the owner of the system of record receiving the user's data indicated in writing that their system notifies or resolves 100% of fatal errors once in their custody? NA

User experience

• [x] Do you capture all of the potential points of failure and make those errors known to the user via email notification and/or through the application on VA.gov or the mobile application?

[holdenhinkle]

Please check if this needs to be done; I'm pretty sure it does:

Does your Datadog monitoring use the appropriate tagging?

• If not, implement tagging standards. Adding the dependency tag is highly recommended!

---

This needs to be done too (see my comments below).

Documentation

• [ ] Do you have a diagram of the submission path that user data your application accepts takes to reach a system of record?
  • If not, then create a user data flow diagram that captures this information.
  • NOTE FROM HOLDEN: Looking at the examples notes here, the diagrams I referenced above are not sufficient.
• [ ] Do you understand how the error is handled when each system in the submission path fails, is down for maintenance, or is completely down?
  • If not, then create documentation that captures how errors in each system are handled. Detail which systems retry a submission and what happens when those retries exhaust. Show this in your diagram.
  • NOTE FROM HOLDEN: I should document our error handling for Find A Rep and the POA widget

[holdenhinkle]

One you complete the above items, please comment in the #zero-silent-failures Slack channel and link to the completed/updated checklist in this ticket (or elsewhere if there's a better place to document the checklist).

This is due October 1st.

Thank you so much

[oddball-lindsay]

Priority for next sprint:

1. Ensure existing Datadog monitoring meets the tagging standards - #93851
2. Update Find and Widget technical documentation to meet new platform standards #93852