search

tomasfarias / airflow-dbt-python

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.
https://airflow-dbt-python.readthedocs.io
MIT License
170 stars 35 forks source link

fix: Avoid heavy imports in operator constructor #86

Closed adamantike closed 1 year ago

adamantike commented 1 year ago

After a previous commit [1] improving import time on operator import, this commit fixes the import time issues when instantiating any of the dbt operators.

The main issue still present was the enum imports in the operator constructors, which make DAGs slow to import any time they instantiated one of them.

Profiling can be easily run locally, with the following command:

python -X importtime -c "from airflow_dbt_python.operators.dbt import DbtRunOperator; DbtRunOperator(task_id='test')" 2>import-times.log

[1] https://github.com/tomasfarias/airflow-dbt-python/commit/67c8d7866235e105675bd208659aac2ccc26070e

tomasfarias commented 1 year ago

note-to-self: Keep this trick in mind for future development.

adamantike commented 1 year ago

@tomasfarias, thanks for the quick review and resolution! Glad to see this project well supported and moving forward.

Moreover, it's a common pattern in other Airflow provider packages

Exactly, I looked at some official providers to follow the same pattern, as I thought that could be a concern if a separate module didn't exist until now.