dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.68k stars 1.61k forks source link

[Feature] Exclude unit tests from running when using "dbt test" command #10656

Closed TowardOliver closed 1 week ago

TowardOliver commented 3 weeks ago

Is this your first time submitting a feature request?

Describe the feature

This is somewhat of an extension of a previous issue: https://github.com/dbt-labs/dbt-core/issues/9237

Currently, including "unit_test" in the DBT_EXCLUDE_RESOURCE_TYPES env variable will prevent units from running when using the dbt build, list and clone commands. No other commands are affected by this env variable.

However, I want to extend this to include the dbt test command.

The use case is something that we're hitting currently, where we run dbt on our airflow instances; We do not want to run unit tests on our Prod airflow instances. This is recommended in the dbt docs here: https://docs.getdbt.com/docs/build/unit-tests#when-to-run-unit-tests

Some of our legacy dags / code do not utilise dbt build. Rather, they use dbt run and dbt test as separate commands. I know that there are ways to limit what is run using specific cli commands, but I think an env variable would be much cleaner, and would prevent us from having to change our code in (potentially) multiple places.

Looking at the ticket I posted above, there seems to be an assumption that no one would want to prevent unit tests on a dbt project level, which I think is true. However, as it currently stands, preventing unit_tests on an environment level is not yet 100% achievable across all functions of dbt.

It would be ideal if we could prevent all our dbt commands from running unit tests with a single env variable, that we could set on whatever airflow instances we need.

Describe alternatives you've considered

We could change all our dags to use dbt build instead of dbt run and dbt test separately.

I believe we could also add an argument to our dbt command to something like this: dbt test --select "test_type:data".

With both these cases, we'd have to change the way we use dbt. Ideally, a global exclusion of a resource_type should be applicable to all functions of dbt.

Who will this benefit?

Anyone who wants to prevent unit tests from running on a per environment basis.

Are you interested in contributing this feature?

I am just about to create a PR for this.

Anything else?

No response

dbeatty10 commented 2 weeks ago

Makes sense @TowardOliver 💡

Thanks for opening a PR for this too!

FishtownBuildBot commented 1 week ago

Opened a new issue in dbt-labs/docs.getdbt.com: https://github.com/dbt-labs/docs.getdbt.com/issues/6101