[spike] DR | Investigate whether LH/VBA records can serve as proof of submission to VBMS.

[data-doge]

Value Statement

As a Veteran I want VA.gov's DR team to be able to quickly determine, through a simple database query, whether NOD and SC forms and associated evidence have been successfully submitted to VBMS So that the DR team can focus its efforts on improving the DR forms, instead of wasting its resources coordinating with multiple stakeholders to confirm whether such submissions are successful.

Background Context

Currently, to verify whether appeals and associated evidence have been successfully submitted all the way to VBMS, we have been manually coordinating with other VA stakeholders (Jacob, Bo) to verify that these submissions appear in CMP and VBMS. This process is inefficient and not scalable.

As an alternative to this process, we may be able to rely solely on the submission status of records in VA.gov's database. Specifically, the VBADocuments::UploadSubmission, AppealsApi::NoticeOfDisagreement, AppealsApi::EvidenceSubmission. Every hour, it appears that vets-api polls central mail for the status of document uploads, and updates VBADocuments::UploadSubmissions appropriately. See:

• https://github.com/department-of-veterans-affairs/vets-api/blob/master/lib/periodic_jobs.rb#L140
• https://github.com/department-of-veterans-affairs/vets-api/blob/master/modules/vba_documents/app/models/vba_documents/upload_submission.rb#L83
• https://github.com/department-of-veterans-affairs/vets-api/blob/master/lib/central_mail/service.rb#L44

Meaning, it's very likely that we can rely on these records to tell whether evidence uploads have been successfully submitted to VBMS, instead of asking CMP to manually check. Ideally, we can do the same for appeal submissions, by referencing the status of LH's AppealsApi::NoticeOfDisagreement records.

Tasks

• [ ] Investigate whether the records above can serve as definitive proof of successful appeal/evidence submission to VBMS, by:
  • [ ] Understanding exactly how the statuses of these records are updated.
  • [ ] Confirming, with LH, to what extent we can rely on these records.
  • [ ] Relatedly, figuring out whether LH's validate endpoint relies on these records.
  • [ ] Confirming, with CMP, that LH is correct.

Acceptance Criteria

• [ ] We know whether the status of the records above can serve as definitive proof of successful submission to VBMS.

Definition of Ready

N/A. Ready now.

Definition of Done

• [ ] Acceptance criteria met.

[saderagsdale]

How is the VBMS status in production created?

• VBMS status is based on a status passed to us from CMP. They give us a complete status, and if the completedReason is uploadConfirmed, we know it's been uploaded successfully into VBMS, and we update the status from success.

Is there documentation I can check out?

• The Benefits Intake documentation is currently the best place. We're actually working to update this documentation now and will also add this information to the Decision Reviews documentation. There's a status table that has this to say about VBMS:

• Indicates this document package was successfully uploaded into a Veteran's eFolder within VBMS.

• On average, submissions reach VBMS status within 3 business days; however, processing times vary and some submissions may remain in a Success status for several weeks before reaching a VBMS status.

• Some document packages are worked in VA systems other than VBMS. For these submissions, Success is the final status.

Are there any internal errors in staging or production that would compromise the accuracy of that status? (Examples: Partially stored data, concurrent transactions interfering with one another, yet still returning the status VBMS)

• Not that we are aware of, I don't believe we've seen anything that would lead us to suspect problems like these.

[saderagsdale]

@data-doge Can we post the status of every piece of evidence in DataDog for complete observability?

[anniebtran]

Started looking into this and going to confirm some of these items with Lighthouse, but here are some of my notes so far:

• Statuses of records are updated by periodic jobs — each form NOD/SC/HLR has their own job and then there's a periodic job that updates VBADocuments::UploadSubmission (i.e. evidence submissions for appeals)
• Seems like we should be able to rely on these records as they are returned by the show/GET endpoints that are referenced in the docs

  Status Caching
Due to current system limitations, data for the status attribute for the following endpoints is cached for one hour. GET /higher_level_reviews/{uuid} GET /notice_of_disagreements/{uuid} GET /notice_of_disagreements/evidence_submission/{uuid} GET /supplemental_claims/{uuid} GET /supplemental_claims/evidence_submission/{uuid} The updated_at field indicates the last time the status for a given GUID was updated.


• The endpoints above that “validate” appeal submissions (i.e. the show actions in their controllers) rely directly on the AppealsApi::<appeal_type> records, but for the evidence endpoints that return AppealsApi::EvidenceSubmissions, the status is delegated to the VBADocuments::UploadSubmission, which is the record that is updated by the periodic jobs

[saderagsdale]

Annie will reach out to LH for additional questions.

[anniebtran]

Chatted with Michel from the LH team a bit and confirmed a few things:

• How reliable are the statuses that are returned from the GET endpoints for the different appeals/evidence? Are they the recommended endpoints for polling?

  We don't have any open bugs or tech debit against those, I think they are working fine and in general they are reliable. They read from our tracking database record, so they are not dependent on central mail or really any other downstream service. Yes, those are the endpoints that you could poll against to monitor the decision review's status. We only update the status we store in our tracking record once an our, so your polling interval should probably not be less than an hour

• How confident do we feel about the statuses returned from CMP? Do you know if they've had any issues with getting status updates from services downstream of them?

  I'm not aware of any issues with the raw CMP status that they hold, I don't think I've worked an issue where CMP specifically advised us to refresh our status because they had it incorrect, so nothing like that on any kind of reoccurring basis or anything like that.

He also mentioned that there was an internal table they use to track status changes and can see how appeals move through their statuses on their way to being established appeals.

[saderagsdale]

Will update Eugene next.

[anniebtran]

TODO:

• Confirm with CMP that the endpoint that Lighthouse uses to update statuses is accurate

[anniebtran]

Might not have to reach out to CMP after all because Timothy Wagner actually mentioned (what I suspect is) the same endpoint that Lighthouse uses to retrieve statuses from their system:

From Timothy's email:

Also, are you aware of the “get status” option? This will allow you to see if a submission is successful and should provide you with packetIDs which would eliminate the need for these spreadsheets going back and forth. If problems occur with a submission, then maybe bring it to us and we can work through it. How does that sound?

And the message from Michel in our thread where he shows an example of what their status call to CMP looks like:

I grabbed a staging NOD, 28e7a25d-f87f-4825-9b0d-c8b0fc5fc301 and made a call to fetch the latest status from CMP staging:

irb(main):007:0> CentralMail::Service.new.status("28e7a25d-f87f-4825-9b0d-c8b0fc5fc301")
response_body=
"[[{\"uuid\":\"28e7a25d-f87f-4825-9b0d-c8b0fc5fc301\",\"status\":\"Received\",\"errorMessage\":null,\"lastUpdated\":\"2024-01-29 18:16:45\",\"packets\":[]}]]">

Here's an example of one of our test submissions on staging that has the packet IDs that Timothy mentioned would get returned:

[[{"uuid"=>"b011f71c-d440-4f34-b231-2ff06d2be256",
   "status"=>"Success",
   "errorMessage"=>"",
   "lastUpdated"=>"2024-02-13 17:25:01",
   "packets"=>
    [{"veteranId"=>"16933629",
      "status"=>"Success",
      "completedReason"=>"UploadSucceeded",
      "transactionDate"=>"02/09/2024 21:17:40"}]}]]

Also if we dig into the code, the endpoint seems to be called getStatus which is what he referred to it as in his email: https://github.com/department-of-veterans-affairs/vets-api/blob/master/lib/central_mail/service.rb#L52

Not sure how much more confirmation we need on this as I would've asked him if the getStatus endpoint was the one that we should be using anyway, so if we feel comfortable with moving forward based on all of these notes, I'd say we're good to go.