department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
282 stars 203 forks source link

[spike] 686c | Investigate validation errors #56452

Closed data-doge closed 1 year ago

data-doge commented 1 year ago

Value Statement

As a software developer I want to investigate the effort to resolve validation errors related to dependency claims So that they don't cause claims to fail.


Background Context

We get a number of address and contact info validation errors from BGS's endpoints which prevent us from establishing a claim via BGS. Here are some examples:

The goal of this ticket is to investigate the errors above and gauge the level of effort to resolve them.

Acceptance Criteria

Tasks

Out of Scope

Definition of Ready

Definition of Done

data-doge commented 1 year ago

@saderagsdale

I've had a chance to review the errors listed above and other errors that have been thrown recently. I have a sense of what changes would be necessary to resolve a substantial number of these name and address validation issues:

  1. We need to add a max length of 30 characters for veteran_contact_information.veteran_address.city on the front end, a max length of 35 characters for last_term_school_information.address.address_line1, and a max length of 35 characters for school_information.address.address_line1. This should be very simple to achieve using VA's form system.
  2. We need to change how we represent UK addresses, specifically. Currently, if the veteran provides an international_postal_code (i.e. GBR), we use an IsoCountryCodes class to automatically expand that to "United Kingdom of Great Britain and Northern Ireland", which exceeds the 50 character limit for countries by two characters. We should consult with Linda and Baha. Ideally, we could simply truncate the country to 50 chars without creating any negative impact.
  3. We need to change how we handle names and addresses with special characters (e.g. apostrophes, umlauts). BIS (and perhaps Corp DB) reject them. We could strip them out, replace them with a space, add front end validations preventing veterans from entering them, or advocate for the relevant datastore(s) to accept special characters.
  4. We need to fix some bugs around foreign military addresses. There are instances where a veteran inputs a city of APO, DPO, or FPO (indicating a foreign military address), but is still allowed to indicate a State address, state code, and zip code. BIS rejects this data because it expects a military_postal_code of either AE, AP, or AA for APO, DPO, and FPO addresses. Additionally, when a veteran indicates a foreign military address, our code deletes city from the payload we send to BIS. But BIS throws an error when city is null. We should consult with Linda and Baha to learn what data is expected for military addresses.
  5. We should improve our logging around validation errors. Currently, I need to sift through enormous error messages returned by BIS to understand whether an error was a validation error, and if so, what kind of validation error. This makes it more difficult to assess the kinds of errors we are getting, and their relative proportions. The relevant part of BIS's error messages tend to start with a ORA-* signature. So, it should be possible to extract the relevant portion of those error messages, and log them alongside the raw error message.

I'm going to quickly chat with Linda and Baha before providing a complexity estimate for the tasks above.

data-doge commented 1 year ago

@saderagsdale I haven't gotten a clear answer from Linda or Baha on points 2 and 4 above, but I don't think that blocks a complexity estimate on this work, or block us from ticketing and commencing this work.

My complexity estimates for the points above, each of which can be represented by its own ticket, are:

  1. 2 pts
  2. 2 pts
  3. 3 pts
  4. 5 pts
  5. 2 pts

Which total 14 points. I think this work will be more straightforward than the timeout/outage-related work, so I propose creating tickets for these points, and commencing them this sprint. I can leave a note in https://github.com/department-of-veterans-affairs/va.gov-team/issues/57921 to that effect.

saderagsdale commented 1 year ago

@data-doge Can you drop tickets into the product backlog and tag them with needs-refinement so we can look at them together before starting?

data-doge commented 1 year ago

Here are those tickets:

I'll leave a note in #57921 as well.

saderagsdale commented 1 year ago

@data-doge I think this is good to close now? Reopen it if not.