686c | Investigate discrepancies across null claims reports.

[data-doge]

Issue Description

In https://github.com/department-of-veterans-affairs/va.gov-team/issues/59298, we asked Matt B. to run his SQL query on the following timeframe: July 3rd 12AM UTC to July 9th 11:59PM UTC. The resulting report contained 68 rows, each representing a potential null claim. Matt intended for this report to be overinclusive.

We queried DataDog for failed and successful claim submissions over the same time period, and exported a report for both cases. Had the number of failed claim submissions in our report been less than or equal to the number of potential null claims in Matt B.'s report, we may have been able to move on with RBPS reprocessing, using Matt B.'s reporting SQL, as a basis. But that was not the case.

Here's a summary of what we discovered:

1. There were 46 purported null claims in Matt B.'s report that we (using DataDog) determined were successfully submitted claims.

As stated previously, Matt B. intended for his report to be overinclusive. So this is not surprising. It does, however, underscore the need to figure out how to deal with false positives (purported, but not actual, null claims) during RBPS re-processing.

Another note: of these 46, 6 were listed as "reprocessing needed." This might just reflect the fact that not all successfully submitted claims to RBPS will be automatically processed by RBPS. Linda has raised this with us before. See RBPS Re-Processing Estimates, pg. 7, fn. 6.

2. There were 19 purported null claims in Matt B.'s report that we also determined were null claims.

This is good news. Notably, all 19 map to SubmitForm686cJob and DependentService failures, but not SubmitForm674Job failures. I recall Matt explaining that his report does not include Form 674 failures, so this tracks.

3. There were 3 purported null claims in Matt B.'s report that weren't captured in either of our reports.

Given these are the last 3 claims in Matt B.'s report (when ordered by timestamp), it's quite possible that Matt B.'s report was using PT time and not UTC time.

4. There were 74 failed claim submissions in our reports that were not found in Matt B.'s report.

We logged 101 failures, 82 of which were not found in Matt B.'s spreadsheet. 8 of these 82 were SubmitForm674Job failures, which as explained previously, we wouldn't expect to find in Matt B.'s spreadsheet. So, there are 74 failed claim submissions (stemming from SubmitForm686cJob and DependentService) in our report that we would expect to see in Matt B.'s report.

Here are some possible explanations for the 74:

• VBMS::SubmitDependentsPdfJob failures. Matt's report, as I understand it, hinges on the existence of a PDF in the veteran's VBMS eFolder. If we didn't upload a PDF, and the claim submission failed, then Matt's report wouldn't capture the failure.
• VANotify::ConfirmationEmail#send failures. This method is called in both submission jobs. If it failed, but the claim submission succeeded, we will still log a failure indistinguishable from failures resulting in null claims. This could account for an over-counting on our part. No matter what, we'll need to refine our logging to account for this distinction.
• Issues with Matt B.'s report. Self-explanatory. There's no guarantee that Matt B.'s report is accurate. Which is why we're validating his report.

---

My recommendation: With the above context in mind, the goal of this spike ticket should be to understand each and every discrepancy between Matt B.'s numbers and ours. Separate, follow-up tickets can be created to resolve any issues that become apparent. Some of those tickets will likely require improvements on our end. Other tickets will likely require improvements on Matt B.'s end.

Findings should be shared with Brandi Traylor, Daniel Bennett, Matt Briaotta, and Matt Self. The weekly Tuesday IPT is probably a good forum to share your findings.

Tasks

• [ ] Investigate all discrepancies and identify their causes.
• [ ] Create tickets to resolve any reporting issues with our code that become apparent.
• [ ] Speak with Matt B. about any issues that can only be attributed to his SQL.
• [ ] Share findings with above stakeholders.

Definition of Ready

N/A. Ready to work on now.

Definition of Done

• [ ] Every discrepancy can be explained.
• [ ] Tickets are created to resolve any issues with our reporting.
• [ ] Matt B. is aware of any potential issues with his SQL.
• [ ] Findings are shared with above stakeholders.

---

Tasks

• [ ] What work is necessary for this story to be completed?

Acceptance Criteria

• [ ] What will be created or happen as a result of this story?

  How to configure this issue

• [ ] Attached to a Milestone (when will this be completed?)
• [ ] Attached to an Epic (what body of work is this a part of?)
• [ ] Labeled with Team (product support, analytics-insights, operations, service-design, Console-Services, tools-fe)
• [ ] Labeled with Practice Area (backend, frontend, devops, design, research, product, ia, qa, analytics, contact center, research, accessibility, content)
• [ ] Labeled with Type (bug, request, discovery, documentation, etc.)

[data-doge]

Hey @steele-lm! Here's that ticket I mentioned. @evansmith and @tfink419 have access to the reports referenced above.

[steele-lm]

Tyler waiting on DataDog access.

[tblackwe]

We believe that #56289 will resolve the majority of BGS Service 403 Errors

[tblackwe]

Similarly it is possible that #56289 will also resolve NoMethodError: undefined method [] for nil:NilClass errors

If not, can we update BGS code? if not, who does?

[tblackwe]

@data-doge we think several of the issues will be cleared up by @evansmith PR.

[tblackwe]

Created the following tickets to resolve discovered issues:

64137

64138

64141

64142