Changes to dispatch in dbt v0.20

[jtcohen6]

Hey T-SQL team!

Given that this package exists alongside dispatch, but doesn't dispatch macros itself, I think this will only require a change to the README. Once users and packages have upgraded to the new dispatch syntax in dbt v0.20, they'll want to specify a first-order project config in dbt_project.yml:

 dispatch:
   - macro_namespace: dbt_utils
     search_order: ['tsql_utils', 'dbt_utils']
   - macro_namespace: dbt_date
     search_order: ['tsql_utils', 'dbt_date']
   - macro_namespace: dbt_expectations
     search_order: ['tsql_utils', 'dbt_expectations']

Instead of:

 vars:
   dbt_utils_dispatch_list: ['tsql_utils']
   dbt_date_dispatch_list: ['tsql_utils']
   dbt_expectations_dispatch_list: ['tsql_utils']

Two crucial differences in the new syntax:

• Users must always specify the name of the dispatching package itself within the search order list
• Users should only specify namespaces for packages that they actually have installed. If someone is only using dbt_utils, they should only specify that one.

In the meantime, there is backwards compatibility for the old syntax, so users on old versions of dbt, or using old versions of packages, can still specify those vars.

[dataders]

@jtcohen6 -- thanks for the heads up! you're right that the end-user experience will change, but this will affect our integration tests.

I think making these changes will allow things to work. my questions are:

1. we need to first wait for upstream repos to change, right?
2. is there a way to associate versions of tsql-utils with specific versions of the parent pacakges without explicitly making the parent packages dependencies? I forsee a potentially tedious table in the readme the lists the version correspondance.

https://github.com/dbt-msft/tsql-utils/blob/28822f1fb9f276270f10d35deb039b8ad8f46239/integration_tests/dbt_utils/dbt_project.yml#L19-L20

https://github.com/dbt-msft/tsql-utils/blob/28822f1fb9f276270f10d35deb039b8ad8f46239/integration_tests/dbt_date/dbt_project.yml#L20-L23

https://github.com/dbt-msft/tsql-utils/blob/28822f1fb9f276270f10d35deb039b8ad8f46239/integration_tests/dbt_expectations/dbt_project.yml#L19-L23

https://github.com/dbt-msft/tsql-utils/blob/28822f1fb9f276270f10d35deb039b8ad8f46239/integration_tests/dbt_audit_helper/dbt_project.yml#L20-L21

[jtcohen6]

@swanderz Really good questions.

1. we need to first wait for upstream repos to change, right?

I thought about this, and realized - you could include both syntaxes if you wanted! As soon as you're running dbt v0.19.2 or higher, this should work just fine:

dispatch:
  - macro_namespace: dbt_utils
    search_order: ['tsql_utils', 'tsql_utils_dbt_utils_integration_tests', 'dbt_utils']

vars:
  dbt_utils_dispatch_list: ['tsql_utils', 'tsql_utils_dbt_utils_integration_tests']

If you've installed the latest version of dbt-utils (0.7+), you've got the dispatch syntax in place. If you've installed an older version of dbt-utils (<0.7.) that uses the old dispatch syntax, great, you have the var there to make it work.

But I don't think this will even be an issue, because each of the four packages for which tsql-utils provides compatibility—dbt_utils, audit_helper, dbt_date, dbt_expectations—have already or will soon be releasing a new minor version that:

• supports dbt v0.20.0
• requires dbt v0.20.0, in fact
• uses the new dispatch syntax

So as soon as you're running your integration tests on dbt v0.20.0, and against new compliant package versions, you could make a hard cutover to the new syntax.

2. is there a way to associate versions of tsql-utils with specific versions of the parent pacakges without explicitly making the parent packages dependencies? I forsee a potentially tedious table in the readme the lists the version correspondance.

The code that's actually in tsql-utils will be compatible with all versions of these other packages, no matter what, regardless of whether they use the old or new dispatch syntax. The thing that will differ is what we called out above: the var/config in dbt_project.yml for your integration tests, and for folks installing the package and setting it up to shim.

Again, because there's going to be a clear cutover from pre-v0.20 support to post-v0.20 support in these other packages, I don't think any user should find themselves in the awkward place of running dbt v0.19.1 with a brand-new version of these packages, or running v0.20.0 with an old version of these packages.

So I don't think there's any compatibility matrix for you to document, really. It's more like: If you're using dbt v0.18 or v0.19, use this old syntax (vars). If you're using v0.20 or newer, use this new syntax (dispatch config); and you could even use both, if you really want.

[telenieko]

 dispach:
   - macro_namespace: dbt_utils
     search_order: ['tsql_utils', 'dbt_utils']
   - macro_namespace: dbt_date
     search_order: ['tsql_utils', 'dbt_date']
   - macro_namespace: dbt_expectations
     search_order: ['tsql_utils', 'dbt_expectations']

Carefull with the typo there! dispach -> dispatch

[jtcohen6]

Ah! Thanks for the catch, just edited above