Discover 0781/a silent failures

[lisacapaccioli]

We have confirmed that there are silent failures of 0781, 14 in the last year. And assumed more going back to when it launched.

This thread reveals that 0781 has not had any historical silent failure investigation and remediation work since the date it launched, and that it has failed silently. It needs the "full treatment" that we gave to evidence upload failures -- finding all the old ones, regenerating them, confirming the volume, and getting them to VBA.

The outcome of this phase should be an artifact that shows when logs are available, and what information we have, and volume counts.

The discovery work should solve for these:

0781

• Identifying how many 0781 uploads failed down the silent path since [last remediation date] to 10/08/24 the date the email went out)
• Identifying the Veterans/users that were impacted
• Identifying how many Veterans' claims were impacted
• Report findings to OCTO

TBD

• Identifying why the 0781 uploads failed
• Help define the work needed for fixing issues

0781a

• Identifying how many 0781a uploads failed down the silent path since [last remediation date] to 10/08/24 the date the email went out)
• Identifying the Veterans/users that were impacted
• Identifying how many Veterans' claims were impacted
• Report findings to OCTO

TBD

• Identifying why the 0781a uploads failed
• Help define the work needed for fixing issues

Out of scope for this Epic

• Actual remediation of the 0781 and 0781a forms (use separate ticket for this)
• Email after LH Migration

Notes This will help VBA make decisions on what to do with the data we have. We likely won't be resubmitting anything on our own, directly into the efolders. Why? Because without an EP, no one will know to look at and re-review it. This process will mirror other code yellows in that we'll be creating spreadsheets, delivering documents, and working with VBA to take action.

See: CY3 knowability table and log timeline for an example of the type of information we'll need. Can be a doc, table, diagram, or a combination.

If the total volume is less than 100 or so, VBA OFO may be willing to handle them in the manual upload/emergency failsafe process.

[lisacapaccioli]

@pacerwow @emilytheis - Team Carbs has capacity to do this now. We've assigned to Scott for Sprint 40.

[freeheeling]

Datadog query covering last 15 months returned 14 results for "Submit Form 0781 Retries exhausted", 1 on 8/31 and the others over an ~12 hour period between 1/19 and 1/20, all before the Form0781DocumentUploadFailureEmail was incorporated on 10/8.

Form526JobStatus.where(job_class: 'SubmitForm0781').where.not(status: %w[success retryable_error try])

• 22 results, from 1/20/24 and 8/31/24, 21 on 1/20/24, between 6:19 and 17:57 UTC, and 1 on 8/31/24.

Form526JobStatus.where(job_class: 'SubmitForm0781').where(status: %w[retryable_error try]).where('updated_at < ?', 2.days.ago)

• 900 results, between 3/4/19 and 8/23/22.

195,623 Form526JobStatus.where(job_class: 'SubmitForm0781') records 194,701 Form526JobStatus.where(job_class: 'SubmitForm0781').where(status: 'success') records difference of 922 records