400 Error in F1 submission: no such file or directory @ rb_sysopen solution

[cloudmagic80]

During the deployment of F1, around 3% of the time, we get this error "no such file or directory @ rb_sysopen" which is a 400 error. This error is caused by missing files stored under the vets-api tmp folder. To resolve this problem, we will recreate the user's pdf and json submission if the files are missing so that the S3 bucket can pick up the files. The resubmitted pdf and json will have same user's attributes except the UUID.

Acceptance criteria:

• Transmission of forms to s3 does not fail silently when there is a problem
• Errors will be reported appropriately when they arise because of an issue unrelated to this one
• Errors are displayed to the user when the retry is unsuccessful
• Errors are not displayed to the user when their submit has recovered with a successful retry
• Make sure the Pega S3 bucket receives the PDF and Metadata JSON files when the user submits the F1 form.

Implementation details

• There will only be 1 retry attempt.
• The Retry is trigger only if 'No such file or directory @ rb_sysopen' is logged in the status.
• If the Retry is unsuccessful, we will still get the 400 error 'No such file or directory @ rb_sysopen' , the applicant will see an error message in the UI and the pdf will not be created. There will be no Pega process triggered. We will get a "missing pega status" in datadog.
• If the Retry is successful, the applicant will see a successful message. Pega will process the pdf. In the backend, we will get 2 processes. The 1st process will show that there was a "missing pega status" and the 2nd attempt will show a completed pega status. They will have 2 Unique IDs

[pshahVA]

Hey, I usually don't comment on engineering details, but since @acrollet is out and as part of my silent failure work, I wanted to know if this retry would still be done while the UX is waiting for a response.

[cloudmagic80]

@pshahVA The applicant will see a completed application message if the retry is successful. If the retry doesn't work, then they will see an unsuccessful message. The retry only happens once and no further retries occur, regardless of the outcome of the second attempt.

[baltes]

@cloudmagic80 Thanks for pursuing this but it still needs discussion around the AC and shouldn't be in progress until that's happened,

[baltes]

Moving this to the backlog for more refinement. Let's hit pause on fixes until we can articulate the approach and define our outcomes.

[pshahVA]

Thank you for the clarification @cloudmagic80

[cloudmagic80]

@baltes @andrea-merrill @pshahVA I've updated the implementation details in the ticket description. Thanks

[cloudmagic80]

https://github.com/department-of-veterans-affairs/vets-api/pull/19006/files