etsy / boundary-layer

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform
Apache License 2.0
262 stars 58 forks source link

[bigquery-operator] Remove destination_dataset_table as a required pr… #105

Closed vchiapaikeo closed 3 years ago

vchiapaikeo commented 3 years ago

…operty

We should remove destination_dataset_table as a required property. There are instances like here (ask Victor for this) where we do not want the bigquery script to write its results to a destination table.

The docs mention:

A dotted ``(<project>.|<project>:)<dataset>.<table>`` that, if set, will store the results of the query. (templated)

However, in the case above, we don't want to store the results in a BigQuery table and BigQuery actually fails the script as a result.

https://github.com/apache/airflow/blob/1.10.3/airflow/contrib/operators/bigquery_operator.py#L41-L43