Overview:
This pull request introduces a central orchestration feature to the ETL pipeline in the form of a primary DAG - this DAG handles all incoming POST requests made on airflow's API endpoint. The DAG then extracts the parameters found on the response payload as input variables to be used by other downstream DAGs for the clipping workflow.
Changes Made:
Added a new DAG script primary_DAG.py to manage POST requests, also included a TriggerDagRunOperator operator to trigger all the downstream DAGs sequentially.
Renamed the function fetch_api (as seen in the first PR raised) as download_geodata with modifications done to retrieve input variables stored on the task instance - this update was done on the utility.py script
Added a new function fetch_payload on the utility.py script. This function stores the API response payload as an x_com message, serving as input for downstream DAGs during the execution of triggerRun operator.
Overview: This pull request introduces a central orchestration feature to the ETL pipeline in the form of a primary DAG - this DAG handles all incoming POST requests made on airflow's API endpoint. The DAG then extracts the parameters found on the response payload as input variables to be used by other downstream DAGs for the clipping workflow.
Changes Made:
primary_DAG.pyto manage POST requests, also included aTriggerDagRunOperatoroperator to trigger all the downstream DAGs sequentially.fetch_api(as seen in the first PR raised) asdownload_geodatawith modifications done to retrieve input variables stored on the task instance - this update was done on theutility.pyscriptfetch_payloadon theutility.pyscript. This function stores the API response payload as anx_commessage, serving as input for downstream DAGs during the execution oftriggerRunoperator.