search

department-of-veterans-affairs / va.gov-cms

Editor-centered management for Veteran-centered content.
https://prod.cms.va.gov
GNU General Public License v2.0
99 stars 69 forks source link

Create Helpdesk article re: troubleshooting Forms broken links #10179

Closed jilladams closed 1 year ago

jilladams commented 2 years ago

Description

https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/find-a-va-form/README.md

Create Helpdesk info so they can act as first line for triage.

Before they can participate, we'll need to make sure Forms managers know where to go to ask for that help, as they don't tend to access Slack, and will need some change management assistance around the workflow.

Acceptance Criteria

CMS Team

Please check the team(s) that will do this work.

jilladams commented 2 years ago

KB Audience: people who are logged in on prod.

What do we want the workflow to be?

Forms broken --> Vet emails Forms Mgr --> Forms Mgr posts helpdesk request with details --> Helpdesk triages, and either: fix, or file ticket to PW with report

Next steps: JIll to draft KB content with Daniel's support, schedule follow up with Stefanie, Troy, Daniel to review troubleshooting steps, failure cases, triage workflow, and comms with Forms Mgrs.

Relevant: Daniel's mural --> https://app.mural.co/t/vagov6717/m/vagov6717/1660150431569/27c80eaeeaedfd88c80322ae177b68c503345ba9?sender=u252e7152800e3981889d1910

jilladams commented 2 years ago

@TroyA6 @stefaniefgray Draft below. This is assuming that Helpdesk folks logged into Prod are the audience, and I wedgied a bunch of context into various places, so if it needs to streamline or reorganize, can do. Thought i'd let you take a look before I edit myself in circles. Let me know if you're able to review out of band, or if scheduling a meeting to give you time to discuss is preferable.

(FYI @dsasser @wesrowe . Would appreciate an accuracy pass here, based on what you've both learned lately. Feel free to just edit the comment directly to address any errors.)

Forms issues in VA.gov

What is find a form

< What is find a form, what it do, who for> < What is Forms DB, how is it used>

Veteran problems with Find a form

Veterans may report problems downloading forms from Find a Form. When this occurs, there are 3 typical causes:

  1. Form metadata has changed upstream of VA.gov in the Forms DB
  2. Form pdf filename has changed upstream of VA.gov in the Forms DB
  3. The API that powers some aspects of the form download URLs (vets-api) may be experiencing high latency, that causes download requests to fail

Veterans may experience any of these 3 issues as a prolonged delay after clicking to download form, or as an error in the UI, e.g.

Error 1

forms_error

Error 2

forms-error2

"Bad PDF Link" email

If a Veteran clicks the link to "Email the forms managers", their email client will open to create a draft email with Subject, "Bad PDF link." The email draft includes a link to the PDF where it can be directly downloaded from the Forms DB. Meaning: a veteran will be unblocked in this use case. Forms Manager will receive this "Bad PDF link" email, including the Form name & URL.

(Need to socialize) Forms Managers: To report a Forms error for help

If you receive a "Bad PDF link" email, please copy/paste the contents of the email to the CMS helpdesk support form. The CMS helpdesk will help to triage the report for follow up.

Helpdesk Forms triage / troubleshooting

Step 1: Verify if the report is internal or public

The helpdesk ticket should include enough information to determine if the broken form is from a public web site form search (va.gov/find-forms) vs. the internal VBA forms search on vba.va.gov.

Step 2: Confirm or Rule out filename or metadata change

Changes in the Forms DB will appear immediately in the the Find a Form search results (which don't route through Drupal CMS). However: data on each Form detail page relies on Drupal CMS. This means that a change in the Forms DB may break user functionality in Find a Form, during the window between a Forms DB change and the Forms DB > Drupal data migration.

"Bad PDF link" reports or other Forms issues can occur if the filename or metadata of a PDF has been changed. To check:

  1. Log into Prod CMS: https://prod.cms.va.gov/

  2. Check the Flagged Content Dashboard in Drupal CMS.

  3. Search page for the reported form, to see if Flags column contains any flags, e.g. "Changed filename," "Changed title," "Deleted" Screen Shot 2022-08-10 at 6 08 25 PM

  4. If the reported form(s) do not appear in the view, jump to Step 3.

  5. If so, click the form name to see node view

  6. In the Form node, in right sidebar, under Recent Changes, see a description of any change. This may be enough to know what to do next. Screen Shot 2022-08-10 at 6 07 47 PM

  7. If you need more detail / confirmation, click the Revisions tab & select the two revisions you want to compare

    Screen Shot 2022-08-22 at 4 34 35 PM

  8. Scroll down and click Compare selected revisions

  9. Under Layout choose Split fields

    Screen Shot 2022-08-22 at 4 35 02 PM

  10. See the details of the change.

    Screen Shot 2022-08-22 at 4 36 17 PM

If the filename or file metadata is not updated / problematic, proceed to Step 3.

If a filename has changed:

PDF filename changes may cause broken links. If a filename changes in the Forms DB, the new filename will sync to the Drupal database (backend) around 2am ET. However, the user-facing link in the website will still point to the old filename (which no longer exists), until the next content release runs, typically ~9am. So the front-end URL may be broken for up to 7 hours, or longer if content-release is delayed.

Form managers have been instructed not to make name changes unless absolutely necessary because it causes the form to lose ranking on search engines and can also break links on other parts of VA.gov.

To resolve:

  1. Visit https://github.com/department-of-veterans-affairs/content-build/actions/workflows/content-release.yml to verify if Content Release has deployed successfully. If not, run the workflow to trigger a release. This will update the user-facing link to point to the new filename, preventing new reports of a broken link. Or, visit #cms-support, and @cms-helpdesk to request a content release.
  2. Email the Forms Manager who owns the form (whether NCA, VBA, VHA, VEO). Notify them of the filename change, and request they revert to the old filename. Ask them to notify you as soon as filename is reverted.
  3. When filename has been reverted, that change will sync to Drupal db at 2am via Forms DB migration sync, again. A content-release should run immediately after, to update the user-facing link again, to point to the restored original filename.

If other metadata has changed:

We have seen at least one instance of swapped metadata in the Forms DB causing issues (#9682). If the filename has not changed, but other metadata has changed related to the report received, use the (PW) Forms/Bad PDF link defect issue template to file a bug for Public Websites team review & describe your findings.

Step 3: Rule out vets-api latency

<add context about latency, what is / why> It's possible that vets-api latency may cause delays / failures to download forms. If you do not have Sentry access:

  1. Check #vets-api-latency-issue-aug08 channel in DSVA slack, which may have recent posts confirming a new latency / cycling event. If you see evidence to indicate that's happening, @ mention @wesrowe in that channel to let him know.
  2. If not, use the (PW) Forms/Bad PDF link defect issue template to file a bug for Public Websites team review, and note that you have checked #vets-api-latency-issue-aug08, but have not checked Sentry.

If you do have Sentry access, to determine if a vets-api latency event is happening:

  1. Check the Find Form error Dashboard: http://sentry.vfs.va.gov/organizations/vsp/discover/results/?id=19&project=4&statsPeriod=7d. If Sentry shows a spike in Forms-related errors, visit #vfs-platform-support, use the Platform support workflow, let them know a latency event may be happening, specify the start time and an end time if it appears to have resolve, & request assistance, e.g. https://dsva.slack.com/archives/CBU0KDSB1/p1660066854784409

Other documentation: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/find-a-va-form/README.md#troubleshooting

jilladams commented 2 years ago

Next steps:

  1. Jill to draft in CMS - save as draft - https://prod.cms.va.gov/node/add/documentation_page
  2. Ping here to ask Stefanie / Troy to review in CMS
  3. Publish
  4. After Publish, disseminate Forms Managers and see if we can get them to buy in on the user support form
jilladams commented 2 years ago

@stefaniefgray @TroyA6 https://prod.cms.va.gov/node/49346/edit now contains what I hope is a solid edit of the content here. Let me know if that's good to publish, or please feel free to edit or schedule time with me to talk through questions about it.

jilladams commented 2 years ago

@stefaniefgray @TroyA6 any feedback? If not, I'd like to move this ticket to be assigned to you both to publish / close when you're ready.