search

ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management
https://ccao-data.github.io/data-architecture/
5 stars 3 forks source link

Update `cleanup_dbt_resources` for seeds and fix `location.neighborhood_group.nbhd` type #348

Closed jeancochrane closed 3 months ago

jeancochrane commented 3 months ago

This PR implements two small changes I neglected to include as part of https://github.com/ccao-data/data-architecture/pull/343:

  1. Updating the cleanup_dbt_resources workflow so that it determines which staging schemas to delete based on seeds as well as models
  2. Manually overriding the column type for location.neighborhood_group.nbhd so that it gets parsed as a string rather than an integer

Note that excluding change 1 from #343 didn't actually cause a bug in the cleanup behavior. Since the only seed we have in the project is currently stored under the location schema, and the location schema is shared with other models, the cleanup workflow successfully deleted the schema based on the existence of models under the location schema (evidence). However, eventually we may have schemas for seeds that are not shared by any models, in which case this change will ensure that the cleanup script still works as expected.

See "z_ci_jeancochrane-update-cleanup-dbt-resources-to-handle-seeds_location"."neighborhood_group" for evidence that nbhd now has the correct type, and see this test cleanup workflow run for evidence that cleanup_dbt_resources continues to work correctly after this change.

jeancochrane commented 3 months ago

Closing temporarily to test cleanup_dbt_resources.