Closed brightemahpixida closed 6 months ago
New Commit Update - b82a8aa:
I include a new file airflow.cfg-dev-test.txt
that has all the configurations to execute a DAG-RUN on dev, the current airflow.cfg
file contains config for the prod environment. I also updated the readme
@gannebamm The feedback made on #9 is currently on this commit (along with the ones made here)
Hi @gannebamm
Just wanted to inform you that i've made a couple of adjustments on the DAGs (as seen on my most recent commit). This update was made to address the error you pointed out on your last comment, which i believe was due to all three DAGs running simultaneously which affected the CPU resources on the airflow instance.
What i did now was adjust the execution flow in a way that each DAG runs sequentially, meaning that the non-active DAGs are kept in a backlog while they wait for the current active DAG to finish its execution. I also made sure to start each iteration with the most resource intensive DAG which is the soil-DAG, followed by the radiation and air-temp DAG
@brightemahpixida By merging this PR the other PRs can be closed, right?
Overview: This pull request adds a new DAG (
fetch_gpkg_soil_data_DAG
) to the ETL pipeline - the role of this DAG is centered on:EPSG:4326
coordinatesgeometry
on the clipped output as aEPSG:25832
format.gpkg
fileChanges Made:
fetch_gpkg_soil_data_DAG.py
that fetches and clips the soil datasetdownload_geodata
function that downloads the geopackage file from the thuenen's cloud storage.env.txt
that holds the password details for the cloud storage site - NOTE: please make sure to include the password on this same file and rename it to.env
clip_soil_data
that utilizes the input attributes (the lattitude and longitude), creates a buffer-extent using both input-coordinates and a buffer, converts the buffer-extent to min/max lat-long coordinates and finally clips the downloaded file to a .gpkg format