search

tomasfarias / airflow-dbt-python

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.
https://airflow-dbt-python.readthedocs.io
MIT License
178 stars 36 forks source link

test(dags): Add tests for dbt operators in DAGs #56

Closed tomasfarias closed 2 years ago

tomasfarias commented 2 years ago

Closes #55

This PR adds more integral testing by defining DAGs with Dbt* operators, triggering runs, and checking outputs. We test both the context manager way of defining a dag as well as the TaskFlow API.

As I was working on this I noticed the example DAGs were actually broken 😅. So this PR also fixes them.

tomasfarias commented 2 years ago

We should add a test that includes a on_failure_callback default parameter and a DbtBuildOperator task to see if we can catch #51

codecov[bot] commented 2 years ago

Codecov Report

Merging #56 (dac0760) into master (c4d13d3) will increase coverage by 0.10%. The diff coverage is n/a.

:exclamation: Current head dac0760 differs from pull request most recent head 7806761. Consider uploading reports for the commit 7806761 to get more accurate results

@@            Coverage Diff             @@
##           master      #56      +/-   ##
==========================================
+ Coverage   98.58%   98.69%   +0.10%     
==========================================
  Files           8        8              
  Lines         917      917              
==========================================
+ Hits          904      905       +1     
+ Misses         13       12       -1
Flag Coverage Δ
unittests 98.69% <ø> (+0.10%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
airflow_dbt_python/operators/dbt.py 100.00% <0.00%> (+0.33%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c4d13d3...7806761. Read the comment docs.

tomasfarias commented 2 years ago

Unfortunately, dag tests interact with a lot of Airflow internals that have changed even from version 2.0 to 2.2. Maintaining all 3 versions of Airflow for that set of tests will be too hard, so the older two have been dropped.

Will update the docs to mention examples have been tested with Airflow > 2.2