search

ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management
https://ccao-data.github.io/data-architecture/
5 stars 3 forks source link

Add class checks and other data validation prior to final db export #369

Closed Damonamajor closed 3 months ago

Damonamajor commented 3 months ago

This PR creates 3 of the 4 dbt tests requested in the following issue . The fourth is on hold, due to the fact that there are known valuation discrepancies between iasWorld and PTaxSim. Thus, we may want to include a buffer on the matching function.

I can't link the issue in the development sections since it's in the PTaxSim repository.

dfsnow commented 3 months ago

Looking great! A question for @dfsnow: Do we want these tests to be output to the QC workbook if they fail? Currently they won't be, because A) they're not tagged with a test_qc_* tag and B) the output workbook isn't written to handle tests outside the iasworld schema anyway. If we do want to output these tests to the QC workbook, I can put up a follow-up PR to handle both of these problems; if not, perhaps we should discuss when and how we expect them to run, since they don't really strike me as unit tests either.

No, these shouldn't go to the QC workbooks. For now, I'm just going to run them manually (via GHA) whenever I export the PTAXSIM DB.