Evidence Upload Silent Failures - investigate logged email notice events of files in eFolder

[lisacapaccioli]

A comparison was made of the number of logs indicating an upload failure email notification had been sent and the number of failures identified when comparing form uploads and eFolder contents.

Submission IDs created after the evidence upload failure email service was enabled that require further investigation:

• 2967040 & 3043407
  • The filenames included in logged email notice events appear in the associated eFolders, but with an upload time that is a couple of hours to a day later than the logged times, indicating the files were uploaded after the claims were submitted, perhaps as a result of receiving notification that the files failed to upload (which would imply the notification prompted resubmission)
• 3043328
  • 3 logged email notice events, of which 2 of the obscured filenames appear to be included in the eFolder documents contents; however, those files include an upload date that was a day before the failed upload email was sent (which I'm unable to explain)

Questions

1. Are these normal, expected results?
2. Is event logging and execution of the Form526DocumentUploadFailureEmail occurring as intended, when the SubmitUploads job has exhausted all retries?
3. What other avenues exist to submit evidence post-526ez submission? Could this explain the presence in the eFolder of files that have logged failed transmission events?

For 2967040 & 3043407:

• [ ] confirm whether mail template mentions claim status tool as option for uploading failed documents
• [ ] confirm how files that failed transmission ended up in eFolder after email notice sent (if through Claim Status Tool, can corresponding logging data be obtained to substantiate?)

For 3043328:

• [ ] compare timestamps of DD logs against eFolder document upload times (is it possible files submitted successfully, but job continued running as if they hadn't and sent a erroneous failure email?)
• [ ] confirm how files that failed transmission ended up in eFolder after email notice sent (similar inquiry as above)

Evidence Upload Silent Failures - Compare doc upload failure email count to suspected failures #92914

[freeheeling]

Submission ID 2967040

updated_at: "2024-08-06 00:14:19

Datadog "Form526DocumentUploadFailureEmail notification dispatched" logs form526_submission_id | 2967040 obscured_filename | the_XXXXXXXXXXXX_XX_XXXXXXes.txt supporting_evidence_attachment_guid | 1431ba13-85cf-4576-813f-26acba9f17b0 timestamp | 2024-08-07T01:15:10Z

Form json Uploads: 2 identically named files (nearly identical content – 2 sentences and contact information different) name: with spaces, no underscores attachmentId: "L015" confirmationCode: different than supporting_evidence_attachment_guid in DD log

name: with spaces, no underscores attachmentId: "L023" confirmationCode: matches supporting_evidence_attachment_guid in DD log

Coded Document Types ['L015' => 'Buddy/Lay Statement'] ['L023' => 'Other Correspondence']

eFolder contents: ordered chronologically by uploadDateTime originalFileName: "FIRST_LAST_ID_526.pdf" uploadDateTime: "2024-08-06T00:14:27Z" documentTypeLabel: "VA 21-526EZ, Fully Developed Claim (Compensation)"

originalFileName: with spaces, no underscores uploadDateTime: "2024-08-06T00:15:50Z" documentTypeLabel: "Buddy / Lay Statement"

originalFileName: underscores instead of spaces uploadDateTime: "2023-08-07T17:01:20Z" documentTypeLabel: "Correspondence"

error_message: [{"key"=>"EVSS_15004", "text"=>"We were unable to complete your request because your file type is not a supported format. Try using a different file type such as PDF (unlocked), JPEG, BMP or TXT. ..." }]

Datadog "Begining document upload file to EVSS" logs for submission's associated user_uuid filesize: 48345 timestamp: "2024-08-07T17:01:10.160176Z"

Summary

Two identically named files, but different document types ("Buddy / Lay Statement" and "Correspondence"), were uploaded. It appears that the upload labeled "Correspondence" failed transmission to the eFolder as part of the 526 submission job, but was uploaded with a modified filename (underscores in place of spaces) ~16 hours after the upload failure email notification was dispatched. The "upload file to EVSS" log has a timestamp that coincides with the filename containing underscores, suggesting the "Correspondence" file was uploaded via the Claim Status Tool.

Submission ID 3043407

updated_at: "2024-09-05 13:54:34"

Datadog "Form526DocumentUploadFailureEmail notification dispatched" logs form526_submission_id | 3043407 obscured_filename | 104XX59.txt supporting_evidence_attachment_guid | 38db9374-e3f7-4e0b-8194-e9d1c8f615c0 timestamp | 2024-09-06T14:55:50Z

Form json Uploads: 1 upload name: 104XX59.txt attachmentId: "L023" confirmationCode: matches supporting_evidence_attachment_guid in DD log

Coded Document Types ['L023' => 'Other Correspondence']

eFolder contents: ordered chronologically by uploadDateTime originalFileName: "FIRST_LAST_ID_526.pdf" uploadDateTime: "2024-09-05T13:54:44Z" documentTypeLabel: "VA 21-526EZ, Fully Developed Claim (Compensation)"

originalFileName: 104XX59.pdf uploadDateTime: "2024-09-06T16:44:32Z" documentTypeLabel: "Correspondence"

originalFileName: 104XX59.pdf uploadDateTime: "2024-09-06T16:45:37Z" documentTypeLabel: "Correspondence"

Summary

Two identically named files in eFolder that match the failed upload's filename, except for the extension, whose upload times to the eFolder are 1 minute apart, and ~2 hours after the upload failure email notification was dispatched. No record in Datadog logs of any file for the user_uuid associated with this submission being sent to EVSS using the Claim Status Tool. I'm stuck trying to explain what occurred in this particular situation, how the file failed to upload to VBMS (the eFolder), yet appears, possibly in duplicate, based on the listed filename.

Submission ID 3043328

updated_at: "2024-09-05 13:06:09"

Datadog "Form526DocumentUploadFailureEmail notification dispatched" logs form526_submission_id | 3043328 obscured_filename | P68XXXXXXXXXXXXXXXXXXXXXXXXXXXXns.txt supporting_evidence_attachment_guid | 8460fdd5-5aad-4546-aa1d-b16b5b099ce6 timestamp | 2024-09-06T14:08:11Z

form526_submission_id | 3043328 obscured_filename | P68XXXXXXXXXXXXXXXXXXXX-XX-XXXXXXXXXXXed.txt supporting_evidence_attachment_guid | d3bd4072-1bd2-4904-810e-c46b811611ca timestamp | 2024-09-06T14:07:56Z

form526_submission_id | 3043328 obscured_filename | P68XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXns.txt supporting_evidence_attachment_guid | 7481edf9-e665-44d1-b1fa-ee76a2b78032 timestamp | 2024-09-06T14:07:30Z

Form json Uploads: name: P68XXXXXXXXXXXXXXXXXXXXXXXXXXXXns.txt attachmentId: "L015" confirmationCode: matches supporting_evidence_attachment_guid in DD log

name: P68XXXXXXXXXXXXXXXXXXXX-XX-XXXXXXXXXXXed.txt attachmentId: "L015" confirmationCode: matches supporting_evidence_attachment_guid in DD log

name: P68XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXns.txt attachmentId: "L015" confirmationCode: matches supporting_evidence_attachment_guid in DD log

Coded Document Types ['L015' => 'Buddy/Lay Statement']

eFolder contents: ordered chronologically by uploadDateTime originalFileName: "FIRST_LAST_ID_526.pdf" uploadDateTime: "2024-09-05T13:06:15Z" documentTypeLabel: "VA 21-526EZ, Fully Developed Claim (Compensation)"

originalFileName: P68XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXns.pdf uploadDateTime: "2024-09-05T13:23:54Z" documentTypeLabel: "Correspondence"

originalFileName: P68XXXXXXXXXXXXXXXXXXXX-XX-XXXXXXXXXXXed.pdf uploadDateTime: "2024-09-05T13:38:57Z" documentTypeLabel: "Correspondence"

originalFileName: P68XXXXXXXXXXXXXXXXXXXXXXXXXXXXns.pdf uploadDateTime: "2024-09-05T13:39:29Z" documentTypeLabel: "Correspondence"

Datadog "Begining document upload file to EVSS" logs for submission's associated user_uuid filesize: 128395 timestamp: "2024-09-05T13:23:47.655062Z"

filesize: 123290 timestamp: "2024-09-05T13:38:50.936267Z"

filesize: 73335 timestamp: "2024-09-05T13:39:22.924899Z"

Summary

Filenames for all 3 logged email dispatch events appear in the eFolder documents contents. However, those files have an upload date that is almost a day before the failed upload email was sent, but 20 minutes after their 526 form was uploaded to the eFolder. The "Correspondence" documentTypeLabel for each of these eFolder files also does not match the "Buddy/Lay Statement" attachment ID of the failed upload. Likely, after submitting their claim, the user visited the Claim Status Tool and, upon reviewing their claim, noticed 3 of the files they uploaded were missing, so they uploaded these files within the Claims Status Tool.

I attempted to confirm this theory by seeking answers to questions posted in the comment following this one. Based on the responses I received, I also did some exploratory investigation in an attempt to determine which files had been been uploaded using the Claim Status Tool. Unfortunately, filesize is the only attribute provided in the payload and does not appear to match the Content-Length of files stored in S3. However, each timestamp is 7 seconds before each suspected corresponding eFolder documents' uploadDateTime, indicating these files were almost certainly uploaded via the Claim Status Tool.

Provided this analysis, it appears that, in some cases, Veterans are visiting the Claim Status Tool soon after submitting their claim, and uploading evidence that was not successfully transmitted to VBMS (the eFolder). Then, after all retries have exhausted on the initial job for all evidence uploaded as part of the form 526 submission, an email (or emails) are sent notifying the Veteran that an upload failed to submit. This raises a couple of questions:

1. Might there be confusion, given this particular scenario, where email notice can take at least 24 hours from the time a claim is submitted, if a Veteran receives a failed evidence upload email after they've corrected for the initial submission failure by uploading the same evidence via the Claim Status Tool?
2. How can a particular file fail upload to EVSS and VBMS via the form 526 flow, yet ultimately be successful submitting to the eFolder, presumably using Claim Status Tool, if both are using the Documents Services API?

[freeheeling]

Questions answered by Jerek Shoemaker in the #benefits-management-tools channel [message]:

• how soon after a disability claim and any supplementary evidence is uploaded to a Veteran's eFolder before those details would be visible within the tool?

  It depends on how long the claim takes to get to VBMS, but once it’s established in VBMS it should show up in the CST

This question may have interpreted "uploaded" as submitted, when I was referring to the document actually arriving in the eFolder

• which service is used for uploading evidence to a pre-existing claim?

  Right now we are using EVSS, but we will be migrating to LH Benefits Documents

• is the process asynchronous?

  Yes

• is there an option for selecting document type when uploading evidence?

  Yes

• what validations other than file type and file size are uploads subjected to?

  The main one is that we have to be able to unlock the PDF if it’s encrypted, other than that the ones you mentioned are the main validations

• do filenames have to be unique?

  No

• Does CST record or log which files have been uploaded using the tool? If so, what data would one need to confirm whether a particular file was uploaded using the tool? Could that be determined via Datadog or the console?

  We don’t currently log anything related to what has been uploaded, though that may change in the future