search

ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management
https://ccao-data.github.io/data-architecture/
5 stars 3 forks source link

Add proper dbt source freshness tests #314

Closed dfsnow closed 5 months ago

dfsnow commented 5 months ago

This PR updates our iasWorld dbt source freshness tests to use an actual loaded_at column (added by https://github.com/ccao-data/service-sqoop-iasworld/pull/10), rather than the date of last change for a record.

It adds freshness checks for all iasWorld tables pulled by sqoop. Since not all tables are updated daily, some checks expect weekly updates.

I can't test this properly until sqoop does a full DB refresh, which will happen tomorrow (2/10), but code review in the meantime would be appreciated!

dfsnow commented 5 months ago

As of this workflow run, all sources are updated with the new loaded_at field and passing checks!