22Q2 Enhancement Wrap Up

[td928]

two reviewers required

see #550 for the full list enhancements completed so far on this dev branch.

Worth mentioning that the new functionalities with DOB now new columns cannot be effectively tested without manually setting the DOB now data date to the versions have the said columns. Similarly for the A2 jobs the DOB version date has to be set separately for fully testing this

One advantage though we have is now we have some new qaqc functionalities especially the output/qaqc_historic.csv. By setting the version to 22Q4 we can have some gut checks using this file against the previous version to see if anything is too out of line which is what I am doing on my local.

for reference is this going to be version.env for local testing

CAPTURE_DATE=2022-11-17
CAPTURE_DATE_PREV=2022-07-05
DOB_DATA_DATE=20220918
VERSION=22Q4
VERSION_PREV=22Q2
VERSION_GEO=22.2.2

[td928]

forgot to merge in the changes from main first and last commit should address that

[mbh329]

Everything ran successfully on my machine! I took a look at the final outputs and intermediate tables to see if they line up with expectations and so far so good! @td928 any specific tests youd recommend besides the qaqc outputs?

[td928]

Everything ran successfully on my machine! I took a look at the final outputs and intermediate tables to see if they line up with expectations and so far so good! @td928 any specific tests youd recommend besides the qaqc outputs?

the qaqc checks reminded me that we have some pretty major upgrades on the main branch see i.e. geo_taxlot. Mine before merging in the changes from the last commit is almost ten times as many flags as previous versions. I am now rerunning with those changes see if it fixes that and some other geographies checks

[mbh329]

Is it worth doing a build via actions off the dev branch with the correct dob data so we can use the qaqc app or does this have implications for housing?

[td928]

Is it worth doing a build via actions off the dev branch with the correct dob data so we can use the qaqc app or does this have implications for housing?

Note here we did this and it turns out the flags count discrepancy were due to the qaqc historic file were based on truncated version for devdb (post 2010 records only) and after that the new version is comparable to the previous versions. This PR should be good to be merged in.