search

ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management
https://ccao-data.github.io/data-architecture/
5 stars 3 forks source link

Upgrade `dbt-athena-community` to `1.7.1` #356

Closed jeancochrane closed 3 months ago

jeancochrane commented 3 months ago

This PR upgrades the dbt-athena-community package (released by the https://github.com/dbt-athena/dbt-athena/ repo) from 1.6.4 to 1.7.1, enabling support for the dbt 1.7.* line.

Note that while 1.7.2 is currently available for dbt-athena-community, I ran into a bug that is preventing it from working with our configuration (https://github.com/dbt-athena/dbt-athena/issues/607). For now, I'm pinning to the second-most recent version, since there don't seem to be any important bugfixes in 1.7.2 according to the changelog and I want to unblock us on upgrading to dbt 1.7.*. I'll upgrade again to 1.7.2 when https://github.com/dbt-athena/dbt-athena/issues/607 is resolved.

jeancochrane commented 3 months ago

@jeancochrane The code changes here look fine, but have we checked to see if there are any breaking changes to the workflows that consume dbt artifacts (manifest.json, catalog.json, etc.). It seems there are some changes to their schemas.

Good note @dfsnow! I had checked the format_dbt_test_failures script but had forgotten we have workflows that read the manifest and catalog. Running them all revealed one permissions issue that I fixed and made permanent here: https://github.com/ccao-data/aws-infrastructure/pull/19 Otherwise, everything looks good to go, so I'm going to go ahead and pull this in.