Closed ireneisdoomed closed 4 months ago
Addition of Google Batch functions using #678 as reference and refactor of the airflow utils we had in common_airflow to interact with the operators into 3 separate modules:
common_airflow
For reference on how to use the Batch interactors, use the variant index DAG https://github.com/opentargets/gentropy/blob/dfae8582cce328b5f91ff9edb3cb360814318943/src/airflow/dags/variant_index.py
utils/batch.py
utils/dataproc.py
check_gcp_folder_exists
read_yaml_config
utils/utils.py
Test run of a DAG
dev
make test
poetry run pre-commit run --all-files
β¨ Context
Addition of Google Batch functions using #678 as reference and refactor of the airflow utils we had in
common_airflow
to interact with the operators into 3 separate modules:For reference on how to use the Batch interactors, use the variant index DAG https://github.com/opentargets/gentropy/blob/dfae8582cce328b5f91ff9edb3cb360814318943/src/airflow/dags/variant_index.py
π What does this PR implement
utils/batch.py
utils/dataproc.py
check_gcp_folder_exists
andread_yaml_config
are moved intoutils/utils.py
π Missing
Test run of a DAG
π¦ Before submitting
dev
branch?make test
)?poetry run pre-commit run --all-files
)?