[Zero Silent Failures SPIKE] Find a VA Form (Lighthouse API connection / migration)

FranECross commented 1 week ago

Description

The following feature needs to be evaluated to determine if it meets the standards for 'zero silent failures'. which is a user-facing transaction that is submitted to the back-end system. If we identify any missing monitoring, etc. from evaluating the checklist, we will file tickets to update implementation.

Find a VA Form:
- Find a Form:
- Form searches to LH Forms API
- LH forms API pulls via sidekiq
- Form Detail Page
- LH forms API pulls via sidekiq
- Forms migration
- LH forms API pulls via sidekiq

OCTODE guidance states:

Does your application have a user-facing transaction that is submitted to a back-end system?
- NOTE: This is not limited to online forms! Other examples can include:
- Uploads of documents and/or attachments
- Performing an action (Such as refilling a prescription or ordering supplies)
Are you using any of the listed APIs?
- Lighthouse Appeals Status
- Lighthouse Benefits Documents API
- Lighthouse Benefits Intake API / Central Mail
- Lighthouse Decision Reviews
- EVSS Document Upload
Does your application submit to an API that relies on Sidekiq (or another background job processor)?

Problem Statement:

See Canvas in Slack

Artifacts

User story

AS A I WANT SO THAT

Engineering notes / background

If you need to set up monitoring in DataDog:

Set up monitoring in Datadog

Follow this guidance on endpoint monitoring to get going. Then following the guidance on monitoring performance to get up to speed with Datadog.

Examples

Benefits Intake API Form Submission Tracking
Banana Peels (Lighthouse Appeals/Forms/Benefits Intake APIs)
Benefits Sidekiq Death Queue view - shows recently exhausted Sidekiq jobs.

Additional examples

Analytics considerations

Quality / testing notes

Acceptance criteria

Checklist

Start

[ ] Do you know when your application shipped to production?
- If not, use Github to determine, roughly, when your application shipped to users.
[ ] Did your application use the same APIs when it shipped as it does today?
- If not, then you'll need to consider the path user data took through both the current architecture and the previous architecture. You will need to account for potential failures in all paths since your application shipped.

Monitoring

[ ] Do you monitor the API that you submit to via Datadog?
- If not, create a ticket to set up monitoring in Datadog.
[ ] Does your Datadog monitoring use the appropriate tagging?
- If not, create a ticket to implement tagging standards. Adding the dependency tag is highly recommended!
[ ] Do errors detected by Datadog go into a Slack notifications channel?
- If not, create a ticket to start directing errors in Datadog to a dedicated Slack channel. See #veteran-facing-forms-notifications for an example.
[ ] Does more than one person look at the Slack notifications channel containing errors on a daily basis?
- If not, create a ticket to then follow this guide on managing errors
[ ] Do the team members monitoring the Slack channel have a system for acknowledging and responding to the errors that appear there?
- If not, create a ticket to then follow this guide on managing errors

⚠️ Failure to have endpoint monitoring in place is a blocking QA standard at Staging review as of 9/10/24. If you answered no to any of the questions above, you will be blocked from shipping at the Staging review touchpoint in Collab Cycle.

Reporting errors

[ ] Have you filed issues for errors that are appearing in Datadog / Slack?
- If not, create a ticket to then start filing Github issues for new categories of errors following this guidance
[ ] Do all fatal errors thrown in your application end up visible to the end user either in the user interface or via email?
- If not, create a ticket to then file Github issues to capture error categories following this guidance

Documentation

[ ] Do you have a diagram of the submission path that user data your application accepts takes to reach a system of record?
- If not, then create a ticket to create a user data flow diagram that captures this information.
[ ] Do you understand how the error is handled when each system in the submission path fails, is down for maintenance, or is completely down?
- If not, create a ticket to then create documentation that captures how errors in each system are handled. Detail which systems retry a submission and what happens when those retries exhaust. Show this in your diagram.
[ ] Has the owner of the system of record receiving the user's data indicated in writing that their system notifies or resolves 100% of fatal errors once in their custody?
- If not, notify @jilladams , @FranECross , @mmiddaugh to work with OCTO to meet with the owner of the system and get their agreement in writing.
- Please document the outcome of this conversation in your product's documentation in Github.

User experience

[ ] Do you capture all of the potential points of failure and make those errors known to the user via email notification and/or through the application on VA.gov or the mobile application?
- If not, note here but don't worry. Few teams are doing this and we'll be providing resources to help you do this in your application. Proceed to create a user data flow diagram. That diagram will help us to help you and your team to create this user experience.
[ ] Create a user data flow diagram
- [ ] Create a ticket to create a user flow based on the requirement noted below:
  - Creating a user data flow diagram is a requirement of the Zero silent errors initiative and will be a required asset at the Architecture Intent touchpoint of the Engineering and Security track of Collaboration Cycle.

Learn how to create a user data flow diagram

File silent errors issues in Github

[ ] Create a ticket to file silent error issues in Github We want to know about your silent errors so that we can help you to fix them. To do this, follow the process in the Managing Errors document.

We don't have any silent errors!

Great! Please let us know that you went through the checklist above as a team and did not find any silent failures in our Slack channel: #zero-silent-failures. You don't have to hang out in there once you have notified us. Just pop in, tell us who you are (which team and in which portfolio) and that no failures were found. Thanks!

jilladams commented 1 week ago

@humancompanion-usds question: Find Forms relies on the LH Forms API, and that API pulls from Drupal via sidekiq job. We are feeling muddy on whether / how that meets the criteria here. We do rely on an API that relies on Sidekiq (or another background job processor), but we do not submit to that API. They pull from us. Curious your thoughts on the ownership of the onus for silent failure, that in mind. Do you happen to know if LH has gotten the same mandate / if they'll be doing similar audits?

humancompanion-usds commented 1 week ago

The most common case of silent failure is when we take data from a user and submit it and that submission fails async. I'm not quite following you you rely on an API that pulls from find a form. If you wouldn't mind pointing me to a data flow diagram, if you have one, that might help me to catch up. There is a template in Mural if you want to create one to demonstrate the flow.

Lighthouse is, for the most part, middleware in that it takes a submission from us and passes it onto another back-end system. When there are failures in those back-end systems Lighthouse returns those to us to manage. To their knowledge they have no errors that they fail to notify us of.

jilladams commented 1 week ago

@humancompanion-usds Sure thing - this is the data flow: https://github.com/department-of-veterans-affairs/va.gov-cms/blob/main/READMES/migrations-forms.md

What happens:

Forms DB (owned by OIT) is the source of truth
Nightly, Drupal pulls Forms DB data and stores it in the Drupal DB. (We use that to create form detail pages for some forms)
Nightly, Lighthouse Forms API then pull that forms data from Drupal, via sidekiq job.
The find-a-form interface queries Lighthouse for Form data.

So the proper user-facing silent failure would be limited to search in Find-a-Form, e.g if a user submitted a search and got no results and no error, or got partial results maybe? We handle that and aren't concerned about that area having silent failures.

But: Michelle is concerned about our data integrations, and making sure that data is making it to its end destinations. We are not really sending data from Find a Form, and if Lighthouse is sure that they don't have errors where they fail to notify, then this ticket might be a no-op?

humancompanion-usds commented 1 week ago

I'd recommend ensuring that we have monitoring in place to ensure we know when the data does not make it to an end destination and that you all have a playbook for how to handle those errors when they pop up in monitoring. This aligns with the endpoint monitoring QA standard.

jilladams commented 5 days ago

If we end up needing to talk to Lighthouse about the state of their monitoring, you can find Kristen Brown and Matt Kelly in the #va-forms channel.

chriskim2311 commented 2 days ago

Spoke to Lighthouse folks in va-forms slack: https://dsva.slack.com/archives/CUB5X5MGF/p1727281368978889.

Lighthouse is monitoring the sidekiq jobs and get alerts if the migration job fails. The job continuously tries to migrate for about an hour before being exhausted. The LH team does have monitors in place and gets alerts for failures of this job through their non-VSP LH Datadog monitors.

In terms of FE for users, failures are shown to the end user for various failures(Invalid PDF Link, Invalid PDF accessed, and onDownloadLinkClick error) and monitors will be moved to datadog with Sentry being deprecated as part of this epic: https://github.com/department-of-veterans-affairs/va.gov-cms/issues/18766.

@FranECross @jilladams I don't think there are any other steps needed for this ticket as monitoring is in place on the LH side of things. Let me know if there are any outstanding questions and I can look into them thanks!

department-of-veterans-affairs / va.gov-cms