search

gocardless / airflow-dbt

Apache Airflow integration for dbt
https://pypi.org/project/airflow-dbt/
MIT License
396 stars 62 forks source link

MWAA integration looking for dbt_project.yml in subdirectories #57

Open giacomorognoni opened 2 years ago

giacomorognoni commented 2 years ago

Hello all!

I am trying to integrate MWAA with Airflow-DBT and for some reason when using the DBT run operator, DBT initially looks for the DBT_Project.yml file in the initial dbt directory (/usr/local/airflow/dags/dbt/dbt_project.yml) as defined using the dir variable. However, it then proceeds to search all subdirectories for the dbt_projects.yml file and resulting in the following error:

[2022-02-13 15:42:52,016] {{dbt_hook.py:130}} INFO - Failed to read package: Runtime Error [2022-02-13 15:42:52,016] {{dbt_hook.py:130}} INFO - no dbt_project.yml found at expected path /usr/local/airflow/dags/dbt/snapshots/dbt_project.yml

Any idea why airflow-dbt searches for dbt_project.yml in the snapshots folder after having found it in the parent folder?

Thank you for all your help!!

RafalSiwek commented 2 years ago

Hi @giacomorognoni, using the operator, have you set the profiles_dir parameter with the path to the directory where the dbt_project.yml is located?