Implement radiation data Fetching/Clipping DAG

[brightemahpixida]

Overview: This pull request adds a new DAG (fetch_cdc_radiation_DAG) to the ETL pipeline - the role of this DAG is centered on:

• fetching the radiation CDC data from the provided FTP site and also storing the user inputs from the upstream DAG (primary_DAG) as an argument to be used later by other tasks
• reading the contents of all the netCDF4 data downloaded to a DASK dataframe
• creating a buffer extent using the lat/long inputs provided by the user as well as the 3km buffer
• converting this buffer extent to min/max lat-long coordinates
• Using the lat-long coordinates (lat-min/max and long-min/max) to clip the subsets of the netCDF4 data
• writitng the clipped output as a parquet file

Changes Made:

• Added a new DAG script fetch_cdc_radiation_DAG.py that fetches and clips the radiation data
• Made adjustments to airflow.cfg config file to suit the local environment on the airflow VM
• Added a new conditional to the download_geodata function that explicitly downloads the geodata from the FTP site for only cdc type DAGs (e.g. fetch_cdc_radiation_DAG, fetch_cdc_air_temp_DAG,...etc.)
• Included a new function clip_data that utilizes the input attributes (the lattitude and longitude), creates a buffer-extent using both input-coordinates and a buffer, converts the buffer-extent to min/max lat-long coordinates and finally clips the downloaded netCDF4 geodata using this attributes and writes the output as a parquet file.
• A verify_file_exists function that acts as some sort of idempotent check to prevent our pipeline redownloading the same files on each new iteration/API-trigger.
• A compute_buffer_extent function that creates a buffer extent using the long/lat input from the user
• And finally a convert_buffer_extent function that converts the buffer-extent provided by the compute_buffer_extent function to a min/max lat-long coordinate, which will be used for the clipping workflow.

[gannebamm]

@kuwet2k please take a look at the code and try to understand the changes. If in doubt, ask @brightemahpixida about it.

[brightemahpixida]

New Commit Update - ea30268:

• Adjustments have been implemented in the clip_data function within the utility.py script. Instead of iterating through the downloaded netCDF4 files, we now read them all at once to reduce overhead.
• The method of converting buffer extent coordinates to min/max longitude-latitude attributes has been updated. In this version, we utilize the GDal library for this operation, replacing the previous approach using the pyprog library (as indicated in the prior commit).

Observations:

• Following the recent test run of this update, we can now extract meaningful clippings from the downloaded dataset using sample latitude and longitude inputs. This is a notable improvement compared to the previous version where generating significant output was challenging.

[brightemahpixida]

@kuwet2k @gannebamm this new commit can be used during the code review

[brightemahpixida]

New Commit Update - 4f71a60:

I switched the output format from Parquet to netCDF, indicating that all our clipped results will now be saved in the netCDF4 format.

[gannebamm]

@kuwet2k Please again take a look at the latest changes

[gannebamm]

@brightemahpixida I have started the review today. Is this code already deployed on the deNBI node?

[brightemahpixida]

@brightemahpixida I have started the review today. Is this code already deployed on the deNBI node?

Hi @gannebamm - yes this code has been deployed to the deNBI node; though right now the interface/version you are seeing right now when you visit the airflow UI is what's served on the main branch.

I could switch that to the latest version now so you could test it out

[gannebamm]

@brightemahpixida I have started the review today. Is this code already deployed on the deNBI node?

Hi @gannebamm ... I could switch that to the latest version now so you could test it out

That would be great. Please do this. It will help in testing the dags

[brightemahpixida]

@brightemahpixida I have started the review today. Is this code already deployed on the deNBI node?

Hi @gannebamm ... I could switch that to the latest version now so you could test it out

That would be great. Please do this. It will help in testing the dags

Hi @gannebamm - i've updated the DAGs on the airflow UI, you can check it out :)

Happy to help out if you encounter any problem or assistance with the test

[brightemahpixida]

👍