[Zero silent failures SPIKE] Facility locator / facilities-api

[jilladams]

Description

The following features needs to be evaluated to determine if it meets the standards for 'zero silent failures', which is a user-facing transaction that is submitted to the back-end system.

OCTODE guidance states:

• Does your application have a user-facing transaction that is submitted to a back-end system?
  • NOTE: This is not limited to online forms! Other examples can include:
  • Uploads of documents and/or attachments
  • Performing an action (Such as refilling a prescription or ordering supplies)
• Are you using any of the listed APIs?
  • Lighthouse Appeals Status
  • Lighthouse Benefits Documents API
  • Lighthouse Benefits Intake API / Central Mail
  • Lighthouse Decision Reviews
  • EVSS Document Upload
• Does your application submit to an API that relies on Sidekiq (or another background job processor)?

Problem Statement:

• See Canvas in Slack

Artifacts

• Audit ticket
• Guidance document
• Sitewide Silent failures assessment
• Sitewide Silent Failure Assessment pre-screen

User story

AS A I WANT SO THAT

Engineering notes / background

If you need to set up monitoring in DataDog:

Set up monitoring in Datadog

Follow this guidance on endpoint monitoring to get going. Then following the guidance on monitoring performance to get up to speed with Datadog.

Examples

• Benefits Intake API Form Submission Tracking
• Banana Peels (Lighthouse Appeals/Forms/Benefits Intake APIs)
• Benefits Sidekiq Death Queue view - shows recently exhausted Sidekiq jobs.

Additional examples

Analytics considerations

Quality / testing notes

Acceptance criteria

Checklist

Start

• [x] Do you know when your application shipped to production?
  • MVP and v2 - 2016, v3 - 2017, PPMS/CCP 2018, PPMS/CCP changes 2023 and 2024
• [x] Did your application use the same APIs when it shipped as it does today?
  • facilities_api/v1 (FacilitiesApi::V1) 2016 up to 6/2024
  • facilities_api/v2 (FacilitiesApi::V2) after 6/2024
  • The v1->v2 changed happened due to Lighthouse changing v0->v1. However, there was parity in the request values (some things like services were excluded from the new results and as a result we added another LH API request to gather that data)

Monitoring

• [x] Do you monitor the API that you submit to via Datadog?
  • We monitor facilities_api/v2/va (wrapper that calls Lighthouse)
  • We monitor facilities_api/v2/ccp (wrapper that calls PPMS ProviderLocator, FacilityServiceLocator, PlaceOfServiceLocator)
  • We do not monitor Lighthouse Facility API (Lighthouse monitors) or PPMS endpoints (not on AWS - on Azure)
• [x] Does your Datadog monitoring use the appropriate tagging?
  • For facilities_api/v2/va ✅
  • For facilities_api/v2/ccp (added external to vets-api, there is no dependency tag possible) ✅
  • If not, implement tagging standards. Adding the dependency tag is highly recommended!
• [x] Do errors detected by Datadog go into a Slack notifications channel?
  • All monitors go to #faciities-monitoring
• [x] Does more than one person look at the Slack notifications channel containing errors on a daily basis?
  • If there are any notifications, Eli looks the day of notifications and follows notifications, but also Michelle Middaugh, PO, and Jill Adams, DM check in. Eli will now use emojis in the managing errors document.
  • If not, then follow this guide on managing errors
• [x] Do the team members monitoring the Slack channel have a system for acknowledging and responding to the errors that appear there?
  • Now adhering to pattern from guide below
  • If not, then follow this guide on managing errors

⚠️ Failure to have endpoint monitoring in place is a blocking QA standard at Staging review as of 9/10/24. If you answered no to any of the questions above, you will be blocked from shipping at the Staging review touchpoint in Collab Cycle.

Reporting errors

• [x] Have you filed issues for errors that are appearing in Datadog / Slack?
  • No errors for things that aren't shown to users
  • No errors that have to be resolved by platform or anyone other than our team
  • If not, then start filing Github issues for new categories of errors following this guidance
• [x] Do all fatal errors thrown in your application end up visible to the end user either in the user interface or via email?
  • Errors when user submits to LH (facilities_api/v2/va POST 400 or 500) shown to user - with a "Find VA locations isn’t working right now" error
  • Errors when user submits to CCP/PPMS (facilities_api/v2/ccp/* 400 or 500) is shown to user - with a "Find VA locations isn’t working right now"

Documentation

• [x] Do you have a diagram of the submission path that user data your application accepts takes to reach a system of record?

flowchart TD
    A(("fa:fa-user User on va.gov/find-locations")) --> B("fa:fa-location-arrow  User enters address or selects use current location")
    B --> C("fa:fa-caret-down Selects VA Health/VA Benefits/VA Cemeteries/Vet Centers") & D("fa:fa-caret-down Community Pharmacies in VA's network") & E("fa:fa-caret-down Selects Urgent Care all in-network or Emergency Care all in-network") & E2("Selects Community Providers in VA's network")
    C --> F("Data transmission")
    D --> F
    E --> F
    F -- Success (2xx) --> G("Output shown to user")
    F -- Error (400 or 500) --> H("Error shown to user<br>'Find VA locations isn’t working right now'")
    E2 --> I("Requests services")
    I -- Success --> J("User selects service form combobox")
    J --> F
    I -- Error --> K("Silent failure")

• [ ] Do you understand how the error is handled when each system in the submission path fails, is down for maintenance, or is completely down?
  • We do not identify to the user the difference between vets-api being completely down or Lighthouse or PPMS connections being unsuccessful from vets-api. All result in the error that the service 'Find VA locations' is not working now.
  • If not, then create documentation that captures how errors in each system are handled. Detail which systems retry a submission and what happens when those retries exhaust. Show this in your diagram.
• [ ] Has the owner of the system of record receiving the user's data indicated in writing that their system notifies or resolves 100% of fatal errors once in their custody?
  • I do not know if we have such an understanding with PPMS or Lighthouse
  • If not, work with OCTO to meet with the owner of the system and get their agreement in writing.
  • Please document the outcome of this conversation in your product's documentation in Github.

User experience

• [ ] Do you capture all of the potential points of failure and make those errors known to the user via email notification and/or through the application on VA.gov or the mobile application?

  • Almost all errors are shown to the user (as though the system had an error). There is one outstanding silent error that should possibly be shown as a system error to the user seen in the diagram above.
  • See https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19249#issuecomment-2369617494 for a point of failure that we need to show an error for. I think we should create an issue for this rather than making a silent failure issue ticket below.

• [x] Create a user data flow diagram

Creating a user data flow diagram is a requirement of the Zero silent errors initiative and will be a required asset at the Architecture Intent touchpoint of the Engineering and Security track of Collaboration Cycle.

Learn how to create a user data flow diagram

File silent errors issues in Github

We want to know about your silent errors so that we can help you to fix them. To do this, follow the process in the Managing Errors document.

We don't have any silent errors!

Great! Please let us know that you went through the checklist above as a team and did not find any silent failures in our Slack channel: #zero-silent-failures. You don't have to hang out in there once you have notified us. Just pop in, tell us who you are (which team and in which portfolio) and that no failures were found. Thanks!

[eselkin]

The prime candidates for needing more error notification are: If PPMS is down and someone selects "Community Providers in VA's network" They will not be notified that the "Service type" will not be populated with proper data so even if they enter data it will complain that they are not selecting something even though it's the system not providing options for them to select.

No network data will be transmitted to vets-api/PPMS when they type some option for "Service type" but don't select anything because a valid selection needs to be made to submit. However, the user will not be aware of this.

/facilities_api/v2/ccp/services does not have its own controller so it can't be monitored directly on datadog, unless we add some properties to vets-api to identify it

[eselkin]

I created this issue: https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19332