[Form 0781/A Failure Mailer] Spike Splitting Form 0781/a

[NB28VT]

Overview

When a veteran fills out Form 526 online, if they filled out the PTSD and or PTSD/assault portions of the form, we submit a Form 0781 and/or Form 0871a form respectively.

If either of these fail, we need to mail and notify them.

However, it was discovered when starting this work that it is likely impossible to implement the mailer properly with the Submit 0781 job in its current state. That, is, the current job handles submitting both Form 0781 and/or a Form 0781a. If we want to properly mail the veteran when either of these actions fail, they will have to be split into separate jobs. In addition, for tracking maintenance purposes these actions should be separate jobs anyway

Acceptance Criteria:

• [x] Validate there is no way to implement a mailer without splitting this job
• [x] Look at historical data in datadog: how often does this fail? (won't describe full scope of problem, no easy way to know what percentage of these were successfully uploaded)
• [x] Scope the work involved in splitting this into two jobs
• [x] Evaluate the risk involved in rewriting and splitting the jobs
• [x] Evaluate the amount of effort that would be required to do this work
• [x] Generate artifact (report) that lays out the need for this in a non technical way, plus the expected time commitment

[NB28VT]

Findings summarized below:

Overview

• We generate PDfs of forms 0781 and 0781a if a veteran fills out the PTSD or PTSD + assault portions of the Form 526 application, respectively.
• Form 0781 and/ Form 0781a submissions are sent to EVSS in separate API calls.
• However, these actions are coupled in the same job in codebase, meaning if either fails to upload to EVSS and retries until we give up, we don't know which one.
• We are planning on mailing the veteran if there is a problem uploading Form 0781, and mailing the veteran if there is a problem with Form 0781a.

The Problem

• Because these uploads happen in the same job, there is currently no way to know which mailer to send if there is a failure.
• We could fix this by first splitting the Form 0781 and 0781a submissions into separate jobs.
• This would require some care because this is a somewhat complex part of the codebase.
• Doing so would also require some changes to code used by the back up path.
• I estimate it would take a focused sprint to adequately split up the job and tests and update the backup path code. And perhaps a sprint to review and test adequately.

Worth it?

• There are a lot of advantages of making this change regardless, including cleaner error tracking, easier code to maintain and an easier path to migrating these form submissions to Lighthouse
• However, it is worth noting there is no record of the current job ever failing in DataDog since we've been logging these failures (December 2023)
• It is also worth noting Form 0781 and Form 0781a will be combined into a single form in the future and it will make this problem moot.
• One alternative suggested by @juliepedtke is to not split the job and send a single email now regardless of what form upload failed, with links to fill out both forms manually, as the veteran would know if they need to manually fill out the PTSD or PTSD + assault form based on their situation. I think this is the ideal solution

[lisacapaccioli]

@NB28VT This is very thorough and I think you addressed most of what I would have asked for. I agree with you and Julie that splitting the jobs doesn't seem necessary. By this time next year we will have a combined PDF. I don't feel that the user experience will be compromised if they get an email even if they didn't answer questions that are relevant to 0781a, especially if our data shows we haven't had a failure since the monitor went into place. Let's just do one message and if @juliepedtke feels that it may be useful to tweak the message for a "just in case" scenario that's fine. Julie just let us know if you want to consider changess to the email content. Thanks and great job clarifying this.

[lisacapaccioli]

@NB28VT Shouldn't this one be done? I thought you did this in Sprint 34.