Implement Soil Density data Fetching/Clipping DAG

[brightemahpixida]

Overview: This pull request adds a new DAG (fetch_gpkg_soil_data_DAG) to the ETL pipeline - the role of this DAG is centered on:

• Downloading the soil density GeoPackage data from the Cloud storage site and also storing the user inputs from the upstream DAG (primary_DAG) as an argument to be used later by other tasks
• reading the contents of all the .gpkg data downloaded to using python's GeoPandas library
• creating a buffer extent using the lat/long inputs provided by the user as well as the 3km buffer
• converting this buffer extent to EPSG:4326 coordinates
• Assigning the converted buffer-extents to a bounding-box and initializing the bounding-box as MultiPolgons
• Setting the MultiPolygons to an empty GeoDataFrame, this will in turn be used to clip the data on step 2
• Executing the clipping workflow using the MultiPolygon dataframe and the downloaded GeoPackage file
• Rewriting back the geometry on the clipped output as a EPSG:25832 format
• writing the clipped output as a .gpkg file

Changes Made:

• Added a new DAG script fetch_gpkg_soil_data_DAG.py that fetches and clips the soil dataset
• Added a new conditional to the download_geodata function that downloads the geopackage file from the thuenen's cloud storage
• Included an environment variable .env.txt that holds the password details for the cloud storage site - NOTE: please make sure to include the password on this same file and rename it to .env
• Included a new function clip_soil_data that utilizes the input attributes (the lattitude and longitude), creates a buffer-extent using both input-coordinates and a buffer, converts the buffer-extent to min/max lat-long coordinates and finally clips the downloaded file to a .gpkg format

[brightemahpixida]

New Commit Update - b82a8aa:

I include a new file airflow.cfg-dev-test.txt that has all the configurations to execute a DAG-RUN on dev, the current airflow.cfg file contains config for the prod environment. I also updated the readme

[brightemahpixida]

@gannebamm The feedback made on #9 is currently on this commit (along with the ones made here)

[brightemahpixida]

Hi @gannebamm

Just wanted to inform you that i've made a couple of adjustments on the DAGs (as seen on my most recent commit). This update was made to address the error you pointed out on your last comment, which i believe was due to all three DAGs running simultaneously which affected the CPU resources on the airflow instance.

What i did now was adjust the execution flow in a way that each DAG runs sequentially, meaning that the non-active DAGs are kept in a backlog while they wait for the current active DAG to finish its execution. I also made sure to start each iteration with the most resource intensive DAG which is the soil-DAG, followed by the radiation and air-temp DAG

[gannebamm]

@brightemahpixida By merging this PR the other PRs can be closed, right?