search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
596 stars 152 forks source link

[Bug] #1102

Open Yaadto opened 2 months ago

Yaadto commented 2 months ago

Astronomer Cosmos Version

Other Astronomer Cosmos version (please specify below)

If "Other Astronomer Cosmos version" selected, which one?

1.5.0

dbt-core version

1.7.6

Versions of dbt adapters

No response

LoadMode

AUTOMATIC

ExecutionMode

AWS_EKS

InvocationMode

None

airflow version

2.7.3

Operating System

-

If a you think it's an UI issue, what browsers are you seeing the problem on?

No response

Deployment

Astronomer

Deployment details

No response

What happened?

when i am using env_var() in the packages.yml and using the env_vars in the ProjectConfig everything is ok . but when i am using the var() in the packages.yml and using the dbt_vars in the ProjectConfig , i get broken dag with error : Compilation Error Could not render {{ var('test').key }}: Required var 'test' not found in config: Vars supplied to = {}

it seems dbt deps doesn't take into account dbt vars ATM: https://github.com/astronomer/astronomer-cosmos/blob/d316b7a051044632b97dfac13360aae4a7fb942c/cosmos/dbt/graph.py#L454-L463

Relevant log output

No response

How to reproduce

using the var() in the packages.yml and using the dbt_vars in the ProjectConfig .

Anything else :)?

No response

Are you willing to submit PR?

Contact Details

No response

dosubot[bot] commented 2 months ago

Hey there, @Yaadto! I'm here to assist you with any bugs, questions, or contributions. Let me know how I can help!

Using env_var() works fine because it directly fetches environment variables at runtime, which are readily available and do not require any pre-compilation or additional context. However, var() in packages.yml and dbt_vars in the ProjectConfig need to be resolved during the compilation phase of dbt. If they are not properly defined or passed, it can result in a broken DAG and compilation errors.

In Cosmos, env_vars are used for both rendering and execution, while dbt_vars are only supported for rendering when using specific load modes like LoadMode.DBT_LS and LoadMode.CUSTOM. This distinction further explains why env_var() might work seamlessly while var() and dbt_vars could cause complications [1][2].

To ensure that dbt deps correctly takes dbt_vars into account, you need to set RenderConfig.dbt_deps to True and define dbt_vars in the project configuration. Additionally, you can pass dbt_vars through the operator_args dictionary when instantiating DbtDag or DbtTaskGroup. This allows you to supply variables to the project, which will override variables defined in the dbt_project.yml [3][4].

Here is an example of setting dbt_vars in operator_args:

DbtTaskGroup(
    # ...
    operator_args={
        "vars": {
            "start_time": "{{ data_interval_start.strftime('%Y%m%d%H%M%S') }}",
            "end_time": "{{ data_interval_end.strftime('%Y%m%d%H%M%S') }}",
        },
    }
)

This should help resolve the issue with dbt deps not taking dbt_vars into account.

To continue talking to Dosu, mention @dosu.