search

department-of-veterans-affairs / va.gov-cms

Editor-centered management for Veteran-centered content.
https://prod.cms.va.gov
GNU General Public License v2.0
98 stars 68 forks source link

Discovery: Forms: review Analytics data to deduce origin server of recent forms outage #10042

Closed jilladams closed 2 years ago

jilladams commented 2 years ago

Description

8/2 we saw a spike in failed PDF downloads from Find a form:

unnamed-2

Acceptance Criteria

CMS Team

Please check the team(s) that will do this work.

dsasser commented 2 years ago

Results:

Domain Count
www.archives.gov 1
www.fiscal.treasury.gov 1
www.opm.gov 1
www.va.gov 38
www.vba.va.gov 70

How I Achieved the Results:

Using Google Analytics results below, I created a local PHP script to query Drupal for the form urls. The script was executed using Drush's script execution 'scr' command.

<?php
$results = [];
$slugs = [
  'about-form-10-10172',
  'about-form-10-10d',
  'about-form-sf180',
  'about-form-26-1880',
  'about-form-21-0966',
  'about-form-20-0995',
  'about-form-21-2680',
  'about-form-21-4138',
  'about-form-21-526ez',
  'about-form-21p-534ez',
  'about-form-40-1330',
  'about-form-10-8678',
  'about-form-10-10ezr',
  'about-form-10-7959c',
  'about-form-22-5490',
  'about-form-10-10ez',
  'about-form-10091',
  'about-form-10-5345',
  'about-form-10-5345a',
  'about-form-20-0996',
  'about-form-21-22',
  'about-form-sf-1199a',
  'about-form-22-1990e',
  'about-form-21-8940',
  'about-form-22-1995',
  'about-form-21-674',
  'about-form-21-686c',
  'about-form-21-22a',
  'about-form-22-5495',
  'about-form-20-10206',
  'about-form-22-0803',
  'about-form-2346a',
  'about-form-21-4192',
  'about-form-27-2008',
  'about-form-5655',
  'about-form-10-10cg',
  'about-form-10-583',
  'about-form-10-7959a',
  'about-form-26-1817',
  'about-form-10-1394',
  'about-form-10-2850c',
  'about-form-10-306',
  'about-form-10-307',
  'about-form-10-7959f-1',
  'about-form-10182',
  'about-form-21-0845',
  'about-form-28-1900',
  'about-form-29-4364',
  'about-form-40-10007',
  'about-form-10-10ec',
  'about-form-10-3542',
  'about-form-21-0781',
  'about-form-21-10210',
  'about-form-21-8764',
  'about-form-21p-8416',
  'about-form-22-1990',
  'about-form-22-6553d-1',
  'about-form-26-8812',
  'about-form-40-0247',
  'about-form-40-1330m',
  'about-form-10-0137',
  'about-form-10-10hs',
  'about-form-21-0781a',
  'about-form-21-0972',
  'about-form-21-4142',
  'about-form-21-4142',
  'about-form-21-674b',
  'about-form-21p-0969',
  'about-form-21p-509',
  'about-form-21p-527ez',
  'about-form-21p-534',
  'about-form-22-8691',
  'about-form-26-4555',
  'about-form-28-0588',
  'about-form-28-8832',
  'about-form-29-0309',
  'about-form-29-336',
  'about-form-29-357',
  'about-form-29-4125',
  'about-form-10-0103',
  'about-form-10-0137a',
  'about-form-10-0459',
  'about-form-10-10d',
  'about-form-10-1394',
  'about-form-10-253',
  'about-form-10-2649a',
  'about-form-10-2850a',
  'about-form-10-7959f-2',
  'about-form-20-0998',
  'about-form-20-572',
  'about-form-21-0304',
  'about-form-21-0779',
  'about-form-21-0788',
  'about-form-21-0789',
  'about-form-21-4502',
  'about-form-21-8760',
  'about-form-21p-0510',
  'about-form-21p-0517-1',
  'about-form-21p-4706b',
  'about-form-21p-527',
  'about-form-22-10271',
  'about-form-22-1999c',
  'about-form-22-5281',
  'about-form-26-0592',
  'about-form-26-1859',
  'about-form-26-8937',
  'about-form-29-0188',
  'about-form-29-0188',
  'about-form-29-8485',
  'about-form-8824i',
  'about-form-fl-10-90',
  'about-form-of-306',
];

foreach ($slugs as $slug) {
  $alias = "/find-forms/{$slug}";
  $db = \Drupal::database();
  $result = $db->select('path_alias', 'pa')
    ->fields('pa', ['path'])
    ->condition('alias', $alias)
    ->execute()->fetchField(0);
  if (is_string($result)) {
    // Get the node id from the result.
    if (str_contains($result, '/node/')) {
      $nid = str_replace('/node/', '', $result);
        // Lookup the form URL for this node.
        $urlResult = $db->select('node__field_va_form_url', 'url')
          ->fields('url', ['field_va_form_url_uri'])
          ->condition('entity_id', $nid)
          ->execute()->fetchField(0);
        if ($urlResult) {
          $results[$nid] = $urlResult;
          print "$nid,$urlResult\r\n";
        }
    }
  }
}

Which output the following:

6157,https://www.va.gov/vaforms/medical/pdf/vha-10-10172.pdf
6046,https://www.va.gov/vaforms/medical/pdf/vha-10-10d-fill.pdf
5803,https://www.archives.gov/veterans/military-service-records/standard-form-180.html
6016,http://www.vba.va.gov/pubs/forms/VBA-26-1880-ARE.pdf
6088,http://www.vba.va.gov/pubs/forms/VBA-21-0966-ARE.pdf
5703,http://www.vba.va.gov/pubs/forms/VBA-20-0995-ARE.pdf
5846,http://www.vba.va.gov/pubs/forms/VBA-21-2680-ARE.pdf
5966,http://www.vba.va.gov/pubs/forms/VBA-21-4138-ARE.pdf
6119,http://www.vba.va.gov/pubs/forms/VBA-21-526EZ-ARE.pdf
5820,http://www.vba.va.gov/pubs/forms/VBA-21P-534EZ-ARE.pdf
6063,https://www.va.gov/vaforms/va/pdf/VA40-1330.pdf
5756,https://www.va.gov/vaforms/medical/pdf/10-8678-fill.pdf
6107,https://www.va.gov/vaforms/medical/pdf/vha-10-10EZR-fill.pdf
5709,https://www.va.gov/vaforms/medical/pdf/VA Form 10-7959c.pdf
5978,http://www.vba.va.gov/pubs/forms/VBA-22-5490-ARE.pdf
5776,https://www.va.gov/vaforms/medical/pdf/10-10EZ-fillable.pdf
5771,https://www.va.gov/vaforms/va/pdf/VA10091.pdf
5763,https://www.va.gov/vaforms/medical/pdf/VA_Form_10-5345_ Fillable.pdf
6137,https://www.va.gov/vaforms/medical/pdf/VHA Form 10-5345a Fill-revision.pdf
5790,http://www.vba.va.gov/pubs/forms/VBA-20-0996-ARE.pdf
5958,http://www.vba.va.gov/pubs/forms/VBA-21-22-ARE.pdf
8005,https://www.fiscal.treasury.gov/files/forms/form-1199a.pdf
5863,http://www.vba.va.gov/pubs/forms/VBA-22-1990e-ARE.pdf
5974,http://www.vba.va.gov/pubs/forms/VBA-21-8940-ARE.pdf
5977,http://www.vba.va.gov/pubs/forms/VBA-22-1995-ARE.pdf
5964,http://www.vba.va.gov/pubs/forms/VBA-21-674-ARE.pdf
6131,http://www.vba.va.gov/pubs/forms/VBA-21-686c-ARE.pdf
5736,http://www.vba.va.gov/pubs/forms/VBA-21-22A-ARE.pdf
5979,http://www.vba.va.gov/pubs/forms/VBA-22-5495-ARE.pdf
5786,http://www.vba.va.gov/pubs/forms/VBA-20-10206-ARE.pdf
6035,http://www.vba.va.gov/pubs/forms/VBA-22-0803-ARE.pdf
6056,https://www.va.gov/vaforms/va/pdf/VA2346a.pdf
5968,http://www.vba.va.gov/pubs/forms/VBA-21-4192-ARE.pdf
5997,http://www.vba.va.gov/pubs/forms/VBA-27-2008-ARE.pdf
6079,https://www.va.gov/vaforms/va/pdf/VA5655.pdf
5804,https://www.va.gov/vaforms/medical/pdf/10-10CG.pdf
5733,https://www.va.gov/vaforms/medical/pdf/vha-10-583-fill.pdf
6074,https://www.va.gov/vaforms/medical/pdf/vha-10-7959a-fill.pdf
6010,http://www.vba.va.gov/pubs/forms/VBA-26-1817-ARE.pdf
5900,https://www.va.gov/vaforms/medical/pdf/10-1394-fill.pdf
6110,https://www.va.gov/vaforms/medical/pdf/vha-10-2850c-fill.pdf
37263,https://www.va.gov/vaforms/medical/pdf/VA_ Form_10-306.pdf
37264,https://www.va.gov/vaforms/medical/pdf/VA_Form_10-307.pdf
5753,https://www.va.gov/vaforms/medical/pdf/vha-10-7959f-1.pdf
6072,https://www.va.gov/vaforms/va/pdf/VA10182.pdf
5872,http://www.vba.va.gov/pubs/forms/VBA-21-0845-ARE.pdf
6069,http://www.vba.va.gov/pubs/forms/VBA-28-1900-ARE.pdf
5991,http://www.vba.va.gov/pubs/forms/VBA-29-4364-ARE.pdf
6053,https://www.va.gov/vaforms/va/pdf/VA40-10007.pdf
6059,https://www.va.gov/vaforms/medical/pdf/10-10EC Fill 2017.pdf
5949,https://www.va.gov/vaforms/medical/pdf/vha-10-3542-fill.pdf
6022,http://www.vba.va.gov/pubs/forms/VBA-21-0781-ARE.pdf
9602,http://www.vba.va.gov/pubs/forms/VBA-21-10210-ARE.pdf
5973,http://www.vba.va.gov/pubs/forms/VBA-21-8764-ARE.pdf
6004,http://www.vba.va.gov/pubs/forms/VBA-21P-8416-ARE.pdf
6118,http://www.vba.va.gov/pubs/forms/VBA-22-1990-ARE.pdf
5792,http://www.vba.va.gov/pubs/forms/VBA-22-6553d-1-ARE.pdf
5984,http://www.vba.va.gov/pubs/forms/26-8812.pdf
6149,https://www.va.gov/vaforms/va/pdf/VA40-0247.pdf
5815,https://www.va.gov/vaforms/va/pdf/VA40-1330M.pdf
5722,https://www.va.gov/vaforms/medical/pdf/VA_Form_10-0137_FILL.pdf
6144,https://www.va.gov/vaforms/medical/pdf/vha-10-10HS-fill.pdf
5740,http://www.vba.va.gov/pubs/forms/VBA-21-0781a-ARE.pdf
5751,http://www.vba.va.gov/pubs/forms/VBA-21-0972-ARE.pdf
5967,http://www.vba.va.gov/pubs/forms/VBA-21-4142-ARE.pdf
5967,http://www.vba.va.gov/pubs/forms/VBA-21-4142-ARE.pdf
6111,http://www.vba.va.gov/pubs/forms/VBA-21-674b-ARE.pdf
5779,http://www.vba.va.gov/pubs/forms/VBA-21P-0969-ARE.pdf
6141,http://www.vba.va.gov/pubs/forms/VBA-21P-509-ARE.pdf
6120,http://www.vba.va.gov/pubs/forms/VBA-21P-527EZ-ARE.pdf
6145,http://www.vba.va.gov/pubs/forms/VBA-21P-534-ARE.pdf
5981,http://www.vba.va.gov/pubs/forms/VBA-22-8691-ARE.pdf
6017,http://www.vba.va.gov/pubs/forms/VBA-26-4555-ARE.pdf
5853,http://www.vba.va.gov/pubs/forms/VBA-28-0588-ARE.pdf
5714,http://www.vba.va.gov/pubs/forms/VBA-28-8832-ARE.pdf
5796,http://www.vba.va.gov/pubs/forms/VBA-29-0309-ARE.pdf
5988,http://www.vba.va.gov/pubs/forms/VBA-29-336-ARE.pdf
5996,http://www.vba.va.gov/pubs/forms/VBA-29-357-ARE.pdf
5990,http://www.vba.va.gov/pubs/forms/VBA-29-4125-ARE.pdf
6114,https://www.va.gov/vaforms/medical/pdf/10-0103-fill.pdf
5983,https://www.va.gov/vaforms/medical/pdf/10-0137A.pdf
5848,https://www.va.gov/vaforms/medical/pdf/10-0459-fill.pdf
6046,https://www.va.gov/vaforms/medical/pdf/vha-10-10d-fill.pdf
5900,https://www.va.gov/vaforms/medical/pdf/10-1394-fill.pdf
11009,https://www.va.gov/vaforms/medical/pdf/VHA Form 253 Ecclesiastical Endorsement for Veterans Affairs Chaplaincy Fill.pdf
5747,https://www.va.gov/vaforms/medical/pdf/10-2649A fill.pdf
6115,https://www.va.gov/vaforms/medical/pdf/vha-10-2850a-091998-fill.pdf
6155,https://www.va.gov/vaforms/medical/pdf/vha-10-7959f-2.pdf
5791,http://www.vba.va.gov/pubs/forms/VBA-20-0998-ARE.pdf
5957,http://www.vba.va.gov/pubs/forms/VBA-20-572-ARE.pdf
5998,http://www.vba.va.gov/pubs/forms/VBA-21-0304-ARE.pdf
6065,http://www.vba.va.gov/pubs/forms/VBA-21-0779-ARE.pdf
6058,http://www.vba.va.gov/pubs/forms/VBA-21-0788-ARE.pdf
5831,http://www.vba.va.gov/pubs/forms/VBA-21-0789-ARE.pdf
5969,http://www.vba.va.gov/pubs/forms/VBA-21-4502-ARE.pdf
6112,http://www.vba.va.gov/pubs/forms/VBA-21-8760-ARE.pdf
5926,http://www.vba.va.gov/pubs/forms/VBA-21P-0510-ARE.pdf
5917,http://www.vba.va.gov/pubs/forms/VBA-21P-0517-1-ARE.pdf
5970,http://www.vba.va.gov/pubs/forms/VBA-21P-4706b-ARE.pdf
6031,http://www.vba.va.gov/pubs/forms/VBA-21P-527-ARE.pdf
18206,http://www.vba.va.gov/pubs/forms/VBA-22-10271-ARE.pdf
5912,http://www.vba.va.gov/pubs/forms/VBA-22-1999c-ARE.pdf
5999,http://www.vba.va.gov/pubs/forms/VBA-22-5281-ARE.pdf
6000,http://www.vba.va.gov/pubs/forms/VBA-26-0592-ARE.pdf
6015,http://www.vba.va.gov/pubs/forms/VBA-26-1859-ARE.pdf
5987,http://www.vba.va.gov/pubs/forms/VBA-26-8937-ARE.pdf
6043,http://www.vba.va.gov/pubs/forms/VBA-29-0188-ARE.PDF
6043,http://www.vba.va.gov/pubs/forms/VBA-29-0188-ARE.PDF
5914,http://www.vba.va.gov/pubs/forms/VBA-29-8485-ARE.pdf
5806,https://www.va.gov/vaforms/va/pdf/VA8824i.pdf
5921,https://www.va.gov/vaforms/medical/pdf/FL 10-90-fill.pdf
5841,http://www.opm.gov/forms/pdf_fill/of0306.pdf
I parsed the domains in the results and generated a pivot table showing the domain and the number of errors reported for the related url: Domain Count
www.archives.gov 1
www.fiscal.treasury.gov 1
www.opm.gov 1
www.va.gov 38
www.vba.va.gov 70

GA Data: analytics_all_va.gov_website_dataproductiontop_events_20220720-20220802.csv

jilladams commented 2 years ago

From scrum: The va.gov files are served as static files from an S3 bucket, so we know availability is not actually the problem. We can deduce that this means the request to fetch the file is breaking down somewhere along the way. We think if users had the actual PDF link, they could receive the file. Suspect: some click tracking event is misfiring and reporting broken links.

When a PDF link reports to the UI that a link is broken, the Veteran gets a modal that says: this link is broken. With an option to email the form manager to report the broken link. Those emails from veterans get a response from a form manager, sending the PDF file.

For bigger outages / full day disconnects between FormsDB / Drupal: Lighthouse API tracks files availability. If it detects a broken link, it will cache the finding for 1 day/24h and report to users in the UI that the link is broken, until the next cycle that the API checks for broken links.