gmyrianthous / dbt-airflow

A Python package that creates fine-grained dbt tasks on Apache Airflow
https://gmyrianthous.github.io/dbt-airflow/
62 stars 23 forks source link

Dynamic manifest data reading #83

Open perttus opened 1 year ago

perttus commented 1 year ago

It would be great to get rid of the static manifest.json file by using dbt ls command or dbt parse or using an Airflow Variable as a storage for the manifest. Otherwise the file needs to be included in the project version control.

gmyrianthous commented 11 months ago

Hi @perttus, thanks for the recommendation.

I would generally agree that manifest is probably not the best way to go, but I am not too sure if dbt ls can do the trick since it will simply return the list of all nodes (without any further details about the nodes themselves or their dependencies).

However I will definitely look into alternatives in the near future such that you don't have to provide a manifest file to render an Airflow DAG.