EqualExperts / dbt-unit-testing

This dbt package contains macros to support unit testing that can be (re)used across dbt projects.
MIT License
413 stars 77 forks source link

Option to not execute on-run-end statements when running unit tests #204

Closed tsaastam closed 7 months ago

tsaastam commented 7 months ago

I have a SQL pipeline I'm converting to use dbt. The pipeline runs in Snowflake and it uses a lot of temporary tables, which are not directly supported by dbt's Snowflake implementation. For this reason I am instead using transient tables (which are relatively cheap in Snowflake, but permanent) and then dropping them at the end of the run. This is done with an on-run-end directive in my dbt_project.yml:

on-run-end:
  - "drop table if exists {{ ref('my_temp_table_1') }};"
  - "drop table if exists {{ ref('my_temp_table_2') }};"

The problem is that when I run my unit tests, dbt also wants to run these drop statements. While I could make this work, it would be much nicer to have an option somewhere to only run the on-run-end statements when I'm not running a unit test - there's no reason to (try to) drop actual tables inside a unit test. How can I accomplish this?

I've discovered a macro, dbt_unit_testing.running_unit_test(), which could in theory be used to only run the drop table statements when the current run is not a unit test:

on-run-end: |
  {% if dbt_unit_testing.running_unit_test() %}
    select 1 from dual;
  {% else %}
    drop table if exists {{ ref('my_temp_table_1') }};
    drop table if exists {{ ref('my_temp_table_2') }};
  {% endif %}

However, the macro doesn't seem to exist at the time when the on-run-end hook is called:

17:15:12.474796 [error] [MainThread]: Encountered an error:
Compilation Error in operation <my_project>-on-run-end-0 (./dbt_project.yml)
  'dict object' has no attribute 'running_unit_test'. This can happen when calling a macro that does not exist. Check for typos and/or install package dependencies with "dbt deps".

Is it possible to configure the dbt-unit-testing package in such a way that I could use this macro from inside dbt_project.yml? Or otherwise, how can I accomplish what I want to do? Is this something that would require additional functionality in dbt itself?

Thank you,

tsaastam commented 7 months ago

Ah, I have actually just discovered how to use the standard flags provided by dbt itself to accomplish what I need. In dbt_project.yml:

on-run-end: |
  {% if flags.WHICH == "test" %}
    select 1 from dual;
  {% else %}
    drop table if exists {{ builtins.ref('my_temp_table_1') }};
    drop table if exists {{ builtins.ref('my_temp_table_2') }};
  {% endif %}

So this issue can be closed.

Perhaps this example could be added to some documentation somewhere, to make it easier for other people to figure out. :)