search

gocardless / airflow-dbt

Apache Airflow integration for dbt
https://pypi.org/project/airflow-dbt/
MIT License
392 stars 62 forks source link

Decouple hook executor from the operator, add Cloud Build execution and adjust testing #45

Closed dinigo closed 2 years ago

dinigo commented 2 years ago

Closes #44

This extracts the execution from the DbtHook. Defaults the Operators behaviour. It does not break the API. But in case the user needs another environment you can pass it a different hook with custom configuration.

dinigo commented 2 years ago

Hello! The functionality is there. We've tested it with a DAG like this

with DAG('dbt_dag_example', start_date=datetime(2021, 10, 1)) as dag:
    # If no hook is provided, it defaults to the local execution hook
    DbtRunOperator(
        task_id='run_dbt_cli',
        project_dir=abspath('./jaffle-shop'),
        profiles_dir=abspath('.'),
        dbt_bin='/home/airflow/.local/bin/dbt',
        # local environment need to define the env variable for auth
        env={ 'GOOGLE_APPLICATION_CREDENTIALS': '/run/secrets/gcp_credentials.json'},
    )
    DbtRunOperator(
        task_id='run_dbt_cloud_build',
        project_dir='jaffle-shop',
        profiles_dir='jaffle-shop',
        dbt_hook=DbtCloudBuildHook(
            project_id='datatonic-uk-dop-dev',
            gcs_staging_location='gs://some_bucket/jaffle-shop.tar.gz',
        )
    )
dinigo commented 2 years ago

@andrewrjones @etoulas : Could you assist me in getting this PR merged? if this repo is not going to be maintained I'll be happy to do so

dinigo commented 2 years ago

This was closed when I renamed the main branch In my repo :/