Discover 0781/a silent failures

[lisacapaccioli]

We have confirmed that there are silent failures of 0781, 14 in the last year. And assumed more going back to when it launched.

This thread reveals that 0781 has not had any historical silent failure investigation and remediation work since the date it launched, and that it has failed silently. It needs the "full treatment" that we gave to evidence upload failures -- finding all the old ones, regenerating them, confirming the volume, and getting them to VBA.

The outcome of this phase should be an artifact that shows when logs are available, and what information we have, and volume counts.

The discovery work should solve for these:

0781

• Identifying how many 0781 uploads failed down the silent path since [last remediation date] to 10/08/24 the date the email went out)
• Identifying the Veterans/users that were impacted
• Identifying how many Veterans' claims were impacted
• Report findings to OCTO

TBD

• Identifying why the 0781 uploads failed
• Help define the work needed for fixing issues

0781a

• Identifying how many 0781a uploads failed down the silent path since [last remediation date] to 10/08/24 the date the email went out)
• Identifying the Veterans/users that were impacted
• Identifying how many Veterans' claims were impacted
• Report findings to OCTO

TBD

• Identifying why the 0781a uploads failed
• Help define the work needed for fixing issues

Out of scope for this Epic

• Actual remediation of the 0781 and 0781a forms (use separate ticket for this)
• Email after LH Migration

Notes This will help VBA make decisions on what to do with the data we have. We likely won't be resubmitting anything on our own, directly into the efolders. Why? Because without an EP, no one will know to look at and re-review it. This process will mirror other code yellows in that we'll be creating spreadsheets, delivering documents, and working with VBA to take action.

See: CY3 knowability table and log timeline for an example of the type of information we'll need. Can be a doc, table, diagram, or a combination.

If the total volume is less than 100 or so, VBA OFO may be willing to handle them in the manual upload/emergency failsafe process.

[lisacapaccioli]

@pacerwow @emilytheis - Team Carbs has capacity to do this now. We've assigned to Scott for Sprint 40.

[freeheeling]

Datadog query covering last 15 months returned 14 results for "Submit Form 0781 Retries exhausted", 1 on 8/31 and the others over an ~12 hour period between 1/19 and 1/20, all before the Form0781DocumentUploadFailureEmail was incorporated on 10/8.

Form526JobStatus.where(job_class: 'SubmitForm0781').where.not(status: %w[success retryable_error try])

• 22 results, from 1/20/24 and 8/31/24, 21 on 1/20/24, between 6:19 and 17:57 UTC, and 1 on 8/31/24.

Form526JobStatus.where(job_class: 'SubmitForm0781').where(status: %w[retryable_error try]).where('updated_at < ?', 2.days.ago)

• 900 results, between 3/4/19 and 8/23/22.

195,623 Form526JobStatus.where(job_class: 'SubmitForm0781') records 194,701 Form526JobStatus.where(job_class: 'SubmitForm0781').where(status: 'success') records difference of 922 records

[freeheeling]

PDF is generated from 0781/a data, then uploaded to EVSS Documents Service, which transfers to VBMS/eFolder.

Within eFolder, documentTypeLabel types, both with originalFileName values that are SecureRandom.hex:

• "VA Form 21-0781, Statement in Support of Claim for PTSD"
• "VA Form 21-0781a, Statement in Support of Claim for Service Connection for Posttraumatic Stress Disorder Secondary to Personal Assault"

*_Verify that claimant's eFolder does not include document type?_**

Available data:

• Form526JobStatus attributes:
  • form526_submission_id
  • error_message
  • updated_at
• associated Form526Submission attributes:
  • user_uuid
  • saved_claim_id
  • submitted_claim_id
  • form_json

Fields to be included in generated CSV:

• date: when final attempt to submit form 0781/a failed
• _job_statusid: 'SubmitForm0781' Form526JobStatus with unsuccessful status
• _submissionid: corresponding to form 0781/a submission
• _saved_claimid: corresponding to form 0781/a submission
• _claim_createdat: SavedClaim 'created_at' timestamp
• _useruuid: Form526Submission 'user_uuid'
• icn: Integeration Control Number
• edipi: Electronic Data Interchange Personal Identifier
• ssn: Social Security Number
• _errormessage: final attempt error message for form 0781/a submission
• _0781_inefolder: boolean indicating whether a file with VA Form 21-0781 documentTypeLabel is present in associated eFolder
• _0781a_inefolder: boolean indicating whether a file with VA Form 21-0781a documentTypeLabel is present in associated eFolder
• _formtypes: indicates whether claim includes 'form0781' data, 'form0781a' data or both

[freeheeling]

'form0781' hash in form json may include:

• nested 'form0781' hash: 401 submissions
• nested 'form0781a' hash: 281 submissions
• both 'form0781' and 'form0781a' nested hashes: 88 submissions
• no nested 'form0781' or 'form0781a' hashes: 152 submissions

[freeheeling]

Draft report of Form 21-0781/a Silent Failure Discovery complete.