ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management
https://ccao-data.github.io/data-architecture/
5 stars 3 forks source link

Deploy dbt model dependencies before building daily models #467

Closed jeancochrane closed 1 month ago

jeancochrane commented 1 month ago

As discussed in https://github.com/ccao-data/aws-infrastructure/pull/26#discussion_r1609011282, there's currently an edge case in our dbt model dependency deployment pipeline that could bite us in the future. The problem is the combination of these three aspects of the pipeline:

As such, it's possible for build-daily-dbt-models to run without model dependencies existing, if they happen to have expired before it runs.

This PR fixes this edge case by updating build-daily-dbt-models to deploy model dependencies in the same way that build-and-test-dbt does.

I tested this PR by manually dispatching build-daily-dbt-models from this branch. (I ran the action just long enough to confirm that the deployment step succeeds, but cancelled it at that point, so the workflow appears to have failed.) Here's evidence that the deployment step succeeds: https://github.com/ccao-data/data-architecture/actions/runs/9193740823/job/25286017600#step:5:28