search

ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management
https://ccao-data.github.io/data-architecture/
6 stars 4 forks source link

Automate Will's QC reports with dbt models and a simple script #525

Closed jeancochrane closed 3 months ago

jeancochrane commented 4 months ago

While we incorporated a number of Will's QC tests into our dbt tests in https://github.com/ccao-data/data-architecture/pull/241, there are a few tests that do not easily map to our current QC testing infrastructure, so Will still ends up manually producing the reports every now and then. We could easily automate these reports by turning the underlying queries that power them into dbt models (probably views, but possibly tables if the queries are expensive), making sure they are tagged with the dedicated test_qc_inquire tag, and then writing a simple Python script that runs them all and outputs them to workbooks.

jeancochrane commented 4 months ago

This isn't quite done, since we still need to migrate over more queries. Opening until we migrate everything to our codebase.