[epic] SC and NOD | Downstream failures

[saderagsdale]

Value Statement

As a Veteran submitting a decision review on VA.gov I want to submit data that will be received by VA without failure or delays So that I can receive my benefits promptly.

Background context

Click to see Slack threads, impact, and error details.

| Error | Count | Summary | Ticket | Status | |---|---|---|---|---| | Maximum page size exceeded. Limit is 78 in x 101 in. | 33 | Most likely long PDFs of web pages. Solution: front end validation and error messages on front-end | https://github.com/department-of-veterans-affairs/va.gov-team/issues/74493 | Done | | Invalid PDF content, part content | 32 | Locked or corrupted PDFs. TBD. | https://github.com/department-of-veterans-affairs/va.gov-team/issues/74503, https://github.com/department-of-veterans-affairs/va.gov-team/issues/79406 | In Progress | | Errors: Batch Submitted with all blank Images.; | 22 | Possible PDF flattening or decryption issue. TBD. | https://github.com/department-of-veterans-affairs/va.gov-team/issues/74511, https://github.com/department-of-veterans-affairs/va.gov-team/issues/79406 | Done | | Unidentified Mail: We could not associate part or all of this submission with a Veteran. Please verify the identifying information and resubmit. | 10 | CMP handles these | https://github.com/department-of-veterans-affairs/va.gov-team/issues/74516 | Done | | Maximum document size exceeded. Limit is 100 MB per document. | 8 | Simple validation update. | https://github.com/department-of-veterans-affairs/va.gov-team/issues/74457 | Done | | Errors: ERR-EMMS-FAILED, Corrupted File detected. EMMS-GCIO-11891026, File: EMMS-GCIO-15229373.xml; | 3 | Strange, screwed up PDFs. Almost certainly not something we can handle programmatically. TBD. | https://github.com/department-of-veterans-affairs/va.gov-team/issues/74521 | Pending | | 412 - Invalid ZIP Code. ZIP Code must be 5 digits, or 9 digits in XXXXX-XXXX format. Specify '00000' for non-US address | 3 | LH fixed this issue a long time ago. | N/A | Done | | Unsupported File type (only PDFs are accepted) | | | | | | Non-string values for keys: zipCode | | | | | | ERR-EMMS-QUARANTINE, Error while processing attached document | | | | | | ERR-EMMS-FAILED, Attachment file does not exist | | | | | | ERR-EMMS-FAILED, Attachment List Count Does Not Match Folder File Count | | | | | - Link to Slack [thread 1](https://dsva.slack.com/archives/C01CJV0L9PS/p1713806179805339)

Desired Outcomes	Success Measures
User: Veterans can submit claims without the consequence of silent failures caused by validation discrepancies.	Decrease % of PDF submissions that fail validation. Decrease % of frontend PDF submission errors that result in form abandonment.
Product: VA.gov eliminates all root causes of silent failures within the service.	Increase % of PDF formats that previously caused silent failures. Decrease % of downstream errors caused by PDFs we accept.
Business: VA does not incur labor and cost consequences associated with validation discrepancies.	Decrease % VA.gov submissions that cause claim delays and unfavorable outcomes.

Out of scope

• Remediation and backup submission processes.

Open questions

• Have we identified all potential errors within scope?
• Have stakeholders removed any errors from the scope?

Tasks

• [x] Confirm if evidence files are living somewhere in an S3 bucket (or elsewhere)
• [x] Compile and send questions to Centralized Mail Portal
• [x] Confirm if Centralized Mail Portal has intervention processes in place for each error type
• [x] Compile and send questions to EMMS/Data Dimensions contact (John Clark)
• [x] Confirm if EMMS/Data Dimensions has intervention processes in place for each error type
• [ ] Define solution for each error type
• [ ] Get status endpoint to check the status of uploaded files
  • [ ] ERR-EMMS-FAILED, Corrupted File detected.
  • [ ] Investigation continuing on “Invalid PDF content, part content”, “Blank images”
• [ ] Get approval on solution approaches
• [ ] Implementation
• [ ] Documentation

  Definition of ready

• [ ] Scope approved by team and stakeholders

Definition of done

• [ ] All potential errors have been identified
• [ ] We know if each system (CMP, EMMS, Data Dimensions) has a process in place to handle each error and make the submission whole
• [ ] Errors are resolved

[saderagsdale]

Linking Emily's comment as a working "definition of done"

[saderagsdale]

Talking to the enablement team today about next steps, as we now know that almost all of these failures are not corrected upstream.

[data-doge]

Summary of downstream errors impacting SC and NOD forms.

(Based on results from LH reports between Oct. 15, 2023 and Jan. 21, 2024)

Error	Count	Summary	Ticket	Status
Maximum page size exceeded. Limit is 78 in x 101 in.	33	Most likely long PDFs of web pages. Solution: front end validation and error messages on front-end	https://github.com/department-of-veterans-affairs/va.gov-team/issues/74493	Validate
Invalid PDF content, part content	32	Locked or corrupted PDFs. TBD.	https://github.com/department-of-veterans-affairs/va.gov-team/issues/74503, https://github.com/department-of-veterans-affairs/va.gov-team/issues/79406	In Progress
Upstream status: Errors: Batch Submitted with all blank Images.;	22	Possible PDF flattening or decryption issue. TBD.	https://github.com/department-of-veterans-affairs/va.gov-team/issues/74511, https://github.com/department-of-veterans-affairs/va.gov-team/issues/79406	In Progress
Upstream status: Unidentified Mail: We could not associate part or all of this submission with a Veteran. Please verify the identifying information and resubmit.	10	CMP handles these	https://github.com/department-of-veterans-affairs/va.gov-team/issues/74516	Done
Maximum document size exceeded. Limit is 100 MB per document.	8	Simple validation update.	https://github.com/department-of-veterans-affairs/va.gov-team/issues/74457	Done
Upstream status: Errors: ERR-EMMS-FAILED, Corrupted File detected. EMMS-GCIO-11891026, File: EMMS-GCIO-15229373.xml;	3	Strange, screwed up PDFs. Almost certainly not something we can handle programmatically. TBD.	https://github.com/department-of-veterans-affairs/va.gov-team/issues/74521	Pending
Upstream status: 412 - Invalid ZIP Code. ZIP Code must be 5 digits, or 9 digits in XXXXX-XXXX format. Specify '00000' for non-US address	3	LH fixed this issue a long time ago.	N/A	Done

[dfong-adh]

Notes on issues and followups: https://docs.google.com/document/d/1FDWa9v1ooG4L__RODnAYZtcqCt4La1azTyZ6vkEr6KA/edit#heading=h.wrbwtwcagj0x

[saderagsdale]

@dfong-adh Hey! Can you drop those in our GitHub space so others can reference them?

[saderagsdale]

Notes from derek's writeup:

Topics of discussion:

• Invalid PDF content, part content

  • Still undergoing investigation on Decision Reviews side as possibly either an issue with the original file that’s uploaded by the Veteran or the decryption step
  • PDF validation endpoint (validate_document) may reduce or eliminate this error
    • Is the updated flow live in production?

• Unidentified Mail

  • The details of these errors, as provided in Lighthouse’s email reports, are: “Downstream status: Unidentified Mail: We could not associate part or all of this submission with a Veteran. Please verify the identifying information and resubmit.” Three months ago, Michael Hobson indicated, based on a old conversation with CMP, that:
    • Answer: "In CMP it is identified as [Unidentified Mail] and then goes through an authorization process where human intervention happens to see if they can track down who the Veteran is. Once all avenues have been take[n] and it is deemed completely unidentifiable it is just stored within CMP"
  • See Slack. See also Slack (“Unidentifiable Mail is handled by people in the same way as with appeals, as we were told previously”). But see Slack (“CMP confirms it does not reach out to the Veteran in case of errors”). But see Slack (“In most cases the file is processed and passed on to VMBS … [e]xceptions being blank files, corrupt files”).
  • It may be good to follow up with CMS if we haven’t done this already. Talk to Sade, and then CMP, to determine whether these errors are already handled by CMP.
    • Seems to be resolved -

• Upstream status: Errors: Batch Submitted with all blank images.

  • “DataDimensions strips out blank pages. If it does that, and then there are no pages left, it returns this error”
  • Still undergoing investigation on Decision Reviews side as either an issue with the original file uploaded by the Veteran or the processing step that decrypts password protected files

• ERR-EMMS-FAILED, Corrupted File detected.

  • Just got this error last night - will need investigation
  • PDF validation endpoint (validate_document) may reduce or eliminate this error
    • Open Question: Is the updated flow using the endpoint live in production?

• ERR-EMMS-QUARANTINE, Error while processing attached document

  • Should be resolved

• ERR-EMMS-FAILED, Attachment file does not exist

  • Should be resolved

• ERR-EMMS-FAILED, Attachment List Count Does Not Match Folder File Count

  • Should be resolved

[saderagsdale]

@dfong-adh Howdy. I tidied up the definition of the epic to reflect some of the details in your documentation. Also added success measures.

[saderagsdale]

Haven't noticed the five errors on the bottom of the list in weeks. Open question:

• Can we close those items?