search

ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management
https://ccao-data.github.io/data-architecture/
5 stars 3 forks source link

Switch dev and staging dbt targets to dedicated staging Spark workgroup #481

Closed jeancochrane closed 3 weeks ago

jeancochrane commented 1 month ago

I noticed during the process of investigating problems with https://github.com/ccao-data/data-architecture/issues/460 that our dbt targets all share the same workgroup for Athena PySpark. This isn't ideal from a permissions perspective, since it means that dev and staging targets can access a role that has CRUD permissions over prod tables and views. This PR changes our profiles so that our dev and staging targets point to a dedicated staging Spark workgroup, primary-spark-staging, that only has CRUD permissions over dev/staging tables and views.

The PR also sneaks in one minor unrelated change: Recently as I've been debugging our builds more extensively, I've been finding myself wishing there was a way that we could manually dispatch a run that rebuilds the entire DAG. As far as I can tell, there is no built-in selector pattern to do this, but I could be wrong. In any case, this PR also adds a feature whereby a single asterisk (*) in the models input will indicate to the build-and-test-dbt workflow that it should rebuild the entire DAG.