Overview:
This pull request focuses on the reworking of the previous DAG implementation within the context of the breedfides-airflow project. The primary emphasis of this PR is on refining the extraction phases of the airflow DAG.
Changes Made:
Added a Makefile with installation commands that streamlines and automates the deployment and configuration processes for the project.
Added a utility.py script to manage extensive tasks and processes external to the DAG configuration file, effectively enhancing its modularity.
Updated the airflow.cfg with new configurations suited for a production environment
Adjusted the examples on the post_example.json for ease in testing
Replaced the previous DAG iterations created by the author of this project
Additional Notes:
To install the dependencies, please execute the command make install-dependencies, I made a couple adjustments to the README - you can easily refer to them during the first initialization of this project on your work-environment.
To enhance the project modularity and ease of management, I intend to create a new branch and subsequent pull request dedicated to managing individual phases of the ETL workflow. Please note that this specific pull request focuses solely on the EXTRACT task for both the WFS and WCS data sources.
NOTE: If you get re-directed to airflow's login page, the username is airflow and the password is airflow
I just merged the updates made on the branch th_primary_dag to this branch, this update has the corrections to the changes requested on this code review
Overview: This pull request focuses on the reworking of the previous DAG implementation within the context of the breedfides-airflow project. The primary emphasis of this PR is on refining the extraction phases of the airflow DAG.
Changes Made:
utility.py
script to manage extensive tasks and processes external to the DAG configuration file, effectively enhancing its modularity.airflow.cfg
with new configurations suited for a production environmentpost_example.json
for ease in testingAdditional Notes:
make install-dependencies
, I made a couple adjustments to the README - you can easily refer to them during the first initialization of this project on your work-environment.