Data Validator Parsing "Bug"

[sthomas93]

When parsing through the Phase 1 & 2 files we noticed that the validator was failing float values when entered and commas for the numeric columns when parsing for syntax errors.

Float values

• 5310018PBI91TYWXY9539V5Y9RN5FM35,20241201,1.00,1.00,988.00,,999.00,,999.00,,999.00,,900.00,0.00,,5.00,20241231,999.00,,999.00,,,,999.00,,,,,,999.00,,999.00,999.00,988.00,,988.00,,988.00,,988.00,988.00,,988.00,988.00
• Do float values have to fail syntax? It's not uncommon on HMDA files to have decimals even if they remain blank. Is this a hard stop when it comes to the creation of a "good" file?

Commas

• Currently can't have numeric values with commas because of the comma separated parsing
• All characters have to be digits allowing for a decimal BUT no comma
• 1,201.00 = Fail
• 1201 = Pass
• What is the level of effort to make this adjustment to the code to not fail for numeric entries that have a comma before the decimal?

Additional Parsing

• Code currently expects exactly "988;988" resulting in "988 ; 988" (space between values) failing
• Should this work this way? If so, then we should make sure the language is clarified in the FIG to prevent this from coming back up.

[jcadam14]

For the Float Values one, this is similar to the "Additional Parsing". Because the FIG calls out values of 1, 988, 999, etc are expected, passing in a value of 1.00 for that field is not allowed. So two are sort of related in that we have very defined terms of what is allowed for certain fields, and while numerical values of 1 and 1.00 are the same, the concept of enumerated lists of acceptable values, it's not.

[jcadam14]

For Commas, it's easy enough to remove the commas when checking if something is a number or is gt, lt, or gte to allowed values. However, that approach would allow someone to put in '9,99,9999,9.00' and simply stripping out commas would give 99999999.00. Which of course the original value should not be accepted as a valid value. This could be achieved easily enough by beefing up the regex that currently just looks for numbers potentially followed by a decimal followed by two numbers.

Regex should look something like: ^\d{1,3}(,\d{3})*(\.\d{2})?$

[sthomas93]

Per team discussion: Validator is parsing as intended to not include commas or floats for data entries. Next steps will be for the BA's to update the corresponding test files and prepare language for Tips & Tricks email.