Discovery: How might MHV on VA.gov alert users about EHR system downtime?

[wesrowe]

Description

User story

As an authenticated user who has access to health tools, I want to be notified in the case that a back-end system has a scheduled or unscheduled downtime window before I have a bad experience.

Notes

• Various documentations available:
  • Platform downtime alerts doc
  • Platform content guidance regarding system alerts
  • PagerDuty maintenance alerts docs
  • Maintenance banner web component
• CMS "full width alert" alternative
  • requires a content release, which typically run every 40-60 mins during business hours
  • looks different from in-page va-alert component
• Solution should also work for the mobile app
• Separate API possibilities are possible: just 1 or all 4 endpoints could go down
  • 3 MHV APIs (rx, records, messaging)
  • 1 VAOS backend
• Oracle (Cerner) doesn't need separate treatment, it and VistA are upstream of the 4 APIs
• Testing DowntimeNotification with yarn mock-api Slack thread

Right after we started this story we learned the tool teams had a PR up for adding downtime alerts to their apps:

• slack thread
• Wes publicized in the thread our intention to recommend a way forward from the MVP solution in the PR

Goals for the solution:

• Can be activated quickly outside of any release or content-build cycles
• Enables responses across all APIs or specific to a single API
  • ...and the alerts can be tailored to apps impacted by the down API(s)
• Ideally would create a single source of truth for API downtime that could be leveraged by the Mobile app

Acceptance criteria

• Documented downtime scenarios for major dependencies that would impact the user experience for MHV on VAgov (tools as well as landing page)
• Drafted recommended approach across landing page and tools

Tasks

• [x] Read all relevant documentation
• [x] Consulted DST (design system team) at office hours
• [ ] Ticketed next step(s), including looping in tool teams (team + Wes)

[wesrowe]

refinement notes:

• MHV API (rx, records, messaging)
• VAOS backend
• Oracle and VistA are farther downstream
• estimate: med

[radavis]

There's an opportunity to refactor the DowntimeNotifications component, removing unused code. The call to getGlobalDowntime() has been commented out for 3 years.

[dcloud]

The presence of the DowntimeNotification component and externalServices object that ties into PagerDuty suggests to me that the scheduled or expected downtime has an existing solution, and next steps, based on Platform Downtime Notifications are:

• Add new PagerDuty services for each MHV "child" service: https://depo-platform-documentation.scrollhelp.site/developer-docs/downtime-notifications#Downtimenotifications-AddinganewPagerDutyservice
• Update devops repo for each
• Have each application add the DowntimeNotification and assign dependencies. The landing page would depend on the "all mhv" service. Apps like medications would depend on the "all mhv" service plus a medications-specific service

It's unclear what steps would be taken for unplanned downtime, aka an outage. PagerDuty certainly has mechanisms for reacting to outages, but I don't know if/how that is set up for VA PagerDuty. I'm not sure if an outage can or should be handled as a maintenance window.

[dcloud]

Began a doc for this at https://github.com/department-of-veterans-affairs/va.gov-team/pull/72520/files?short_path=db315a8#diff-db315a8adad6908236b3c8cd9f4d60ac6a40ac2dbdd57cd9048cdb50523450e5

[wesrowe]

Notes from sync with Sara, Keith, Jim, Wes:

• Tool teams will implement downtime-alert component as SM is already doing
• Eng need to figure out nuts and bolts about how to actually make Appt and MHV Pagerduty flags available to landing page (to go into new ticket)
  • anticipate modifying the existing DT alert code to ease tool teams' adoption
• Content: finalize content
  • for tool content, need to pull SM's content and just get the same message across tools
  • for landing page, adapt that content for for Appts downtime alone, MHV-wide downtime alone, or both at the same time
  • ~[we decided not to alert on the LP when individual tool services are down]~ EDIT: this note was misleading, and I'm not sure what it was meant to convey

[dcloud]

I've drafted a doc in va.gov-team for general findings. It needs work and updates since the MHV app/services teams have merged their PR

[keithcheung828]

Updated Mural for downtime notifications design

[dcloud]

@keithcheung828 We can confirm that the DowntimeNotification component allows us to customize what gets rendered, with the minor caveat that we would need to handle logic around approaching vs current downtime in addition to the logic for what the combination of alerts would look like.

[radavis]

When reading this user story, I began to wonder how we might handle a service outage and wrote out some thoughts.

User Story: As an authenticated user who has access to health tools, I want to be notified in the case that a back-end system has a scheduled or unscheduled downtime window before I have a bad experience.

Unscheduled downtime, as in "a supporting service will be taken down, soon/now" can be handled as a scheduled downtime event, and a corresponding maintenance window for the service can be created within PagerDuty, triggering the DowntimeNotification component.

Unscheduled downtime, as in "a supporting service has become unresponsive," aka, an outage, should be handled 1) gracefully by FE code when CRUDing resources, 2) by the breakers gem on the BE preventing requests to the downstream service and responding with an appropriate HTTP status code, 3) by ops teams, who would see that one or more services have become unresponsive, 4) by team members, who would create a maintenance window within PagerDuty for the downed service as part of the incident response procedure. Note that the breaker status of services can be found in Datadog.

Perhaps there is an opportunity here to have vets-api update PagerDuty when a breaker trips on a service integration, automatically creating a maintenance window.

[dcloud]

FWIW, the maintenance banner says:

This component is ONLY for site-wide system status messages. There is no other appropriate use.

[wesrowe]

We need to carry this story into sprint 22. There seems to be some needed coming together between Engineering and Design. The AC is "Drafted recommended approach across landing page and tools."

• E.g., the question in comments above seems unanswered, of what design system component to use on the landing page.
• I don't think I saw anything in Keith's presentation about the alerts on the tool pages. Daniel found that the Appts team did something custom with the downtimeAlert component. That's a key input to the UX design across the portal; the landing page can't just do our own thing, it needs to rhyme with the tools' implementations.

[keithcheung828]

Consulted with DS team that the downtime alert design is based on the alert page on design systems

[wesrowe]

Some interesting discoveries that will guide our recommendation:

• tool-level implementation from late December has several major UX issues that need to be addressed (slack thread)
  • didn't include time window
  • has no reference to app that is missing from page – even the H1 from the page is gone. this is an a11y issue, so urgent before next downtime.
  • Daniel suggested adding an opt-in header feature to the downtime notification component
• the "all-MHV" PagerDuty service that the tool teams are leveraging may not mean what we thought it meant. Per Stephen Barrs, that PD service "is owned by the identity team and is triggered when there are issues with users attempting to authenticate using the MHV credential." this is a totally separate functional domain from the APIs.
  • so our rec may need to include a new "all MHV" PD service
  • some operational questions were raised in that thread about automation, let's keep in touch with Janie Tankard-Carnock, Kolby Balch (SM eng who did tool DTA implementations), Stephen Barrs, Mark Dewey
• will a modal be involved? the docs of the DTA component say a modal appears in the final hour before a window; however, I haven't seen anything about a modal in our team's investigation so far.

[wesrowe]

recommendation scope from meeting 1/8 (Daniel, Keith, Wes):

• assume a single MHV PagerDuty
• recommend content for landing page and tools (including headings and alert content)
  • recommend heading/tool name for tool pages
  • landing page will warn users about missing functionality but not hide any tool links
• technical recommendation for whether/how DTA component itself needs to be changes to accommodate UX recommendations (for either landing page or tool pages)
• modals required in current DTA component?
  • custom render prop overrides the modal trigger, so it won't show on any tool page (Appts is passing a custom render prop, but other tools use out of box with modal)

[dcloud]

Also in our discussion, it seemed we were focused on timed maintenance windows, aka scheduled downtime. PagerDuty will provide start and end times regardless of whether the maintenance window was planned/scheduled or not.

The platform component doesn't mention start or end times, perhaps assuming that the unscheduled downtime case is more common than the scheduled maintenance window.

[wesrowe]

I told Janie (PO for SM) in this thread that we would share our recommendation there when it's ready. There's another thread where I should post it also when the time comes.

[keithcheung828]

Mural updated to contain:

1. MHV downtime

   

2. Single tool downtime

   

3. Multi tool downtime

   

[keithcheung828]

4. If alerts are not able to specify tools that are down, an alternative design is shown below:

[dcloud]

I merged the discovery doc so its editable on the main branch now: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/health-care/digital-health-modernization/engineering/mhv-downtime-notification-discovery.md

[keithcheung828]

Please see the updated mural link.

[wesrowe]

@keithcheung828, is this screenshot from the mural accurate? It's supposed to be the current state of the Inbox page, in the pre-maintenance timeframe. But it doesn't have an H1.

[wesrowe]

@keithcheung828 made final changes to mural. closing!