Implement radiation data fetching DAG (DWD CDC netCDF)

[gannebamm]

Status Quo

Currently, only some POC DAGs are defined. There will be a meeting with bioinformatic researchers to define which environmental datasets are helpful for their analyses.

Expected behaviour

There is a ready-to-use DAG for the first environmental dataset. The first DAG shall extract data from DWD CDC:

radiation

• metadata: https://opendata.dwd.de/climate_environment/CDC/grids_germany/daily/Project_TRY/radiation_global/DESCRIPTION_gridsgermany_daily_Project_TRY_radiation_global_en.pdf
• FTP URL: https://opendata.dwd.de/climate_environment/CDC/grids_germany/daily/Project_TRY/radiation_global

The datasets are stored at the CDC FTP and in netCDF format. The netCDFs are zipped. The radiation data from 2000 onwards shall be downloaded, the zip extracted and the netCDF clipped to the given lat / long position + 3km radius.

[gannebamm]

We still need information until we can start to further define this issue.

[gannebamm]

@brightemahpixida the first dataset was defined in today's meeting. See above.

There are more CDC netCDF datasets to follow. Those will be part of different issues. To give you some heads-up for the soon following tasks:

• air temperature: https://opendata.dwd.de/climate_environment/CDC/grids_germany/daily/Project_TRY/air_temperature_mean/DESCRIPTION_gridsgermany_daily_Project_TRY_air_temperature_mean_en.pdf
• precipitation: https://opendata.dwd.de/climate_environment/CDC/grids_germany/daily/hyras_de/precipitation/DESCRIPTION_GRD_DEU_P1D_RR_HYRAS_DE_en.pdf
• humidity: https://opendata.dwd.de/climate_environment/CDC/grids_germany/daily/Project_TRY/humidity/DESCRIPTION_gridsgermany_daily_Project_TRY_humidity_en.pdf
• wind speed: https://opendata.dwd.de/climate_environment/CDC/grids_germany/daily/Project_TRY/wind_direction/BESCHREIBUNG_gridsgermany_daily_Project_TRY_wind_direction_de.pdf

PRIO 2:

• wind direction: https://opendata.dwd.de/climate_environment/CDC/grids_germany/daily/Project_TRY/wind_speed/DESCRIPTION_gridsgermany_daily_Project_TRY_wind_speed_en.pdf

[gannebamm]

@brightemahpixida if you need further information regarding the datasets, you can ping @vineetasharma105. She has experience with netCDF files.

[brightemahpixida]

Thanks @gannebamm - i'm currently having a look at this now, i'll make sure to ping Vineeta if i have a question 👍

[brightemahpixida]

Hi @gannebamm, i got a couple of questions on this:

• I assume this new DAG we'll be creating will be included with the WCS and WFS DAGS (which we created on the first PR) or are we starting afresh since the title of this issue states that this is our first DAG?

• Will the extraction of all environmental datasets be consolidated into a single DAG, or will each dataset (e.g., radiation, air temperature, precipitation) have its own independent DAG?

• How often are the data uploaded or updated on the FTP site, i'm looking into maybe including a DAG schedule will need to applied for this use-case because as far as i know this type of DAG will probably not be the same as the other 'api-DAGs' that are triggered by a POST request

• To clarify the data extraction from the FTP site, in one of your previous comments, you mentioned downloading radiation data from 2000 onwards. Does this refer to the year specified in the filename, as seen in examples like SIS_200001_daymean.nc.gz?

[brightemahpixida]

Hi @vineetasharma105 - I also need a bit of clarification on the netCDF file

• I had a look at one of the sample netCDF files, i know we intend on retrieving the latitude and longitude attributes from the variables as opposed the dimensions or the metadata - but just to clarify, are we conducting separate calculations for both the latitude and longitude attributes using the specified radius mentioned?

[vineetasharma105]

Hi Bright, (If I understood your question correctly), the variable value is fetched for a certain 'time' and certain 'Lat-Long' combination. And the Latitude and Longitude is always to be used together (that makes a spatial point) and this will be used to clip based on the specified radius. Please let me know if I was not able to understand your question again.

[gannebamm]

I assume this new DAG we'll be creating will be included with the WCS and WFS DAGS (which we created on the first PR) or are we starting afresh since the title of this issue states that this is our first DAG?

You are right. I will change this issues title. The WCS and WFS DAGs are the first DAGs provided. This one is the second.

[brightemahpixida]

Thanks @vineetasharma105 - I think i now understand it to a degree, but maybe i need a little bit clarification on the 'clip' concept; would you be free or ok with a short call later on today (15:00); i.e. if you are free today, if not we could schedule the call for tomorrow

Or how does that sound to you : )

[vineetasharma105]

@brightemahpixida : Hi Bright. Sorry for the late response. Yes we can have a call today. Let me know how you want to have it.

[brightemahpixida]

Hi @vineetasharma105, thanks for the reply - will 14:00 or 15:00 work for you today, i don't think it's going to be a long session. I can setup a webex link later on once i get your reply

[vineetasharma105]

@brightemahpixida : Yes, this time slot is good.

[brightemahpixida]

@vineetasharma105 Great thanks, i just sent a link now to your mailbox

[brightemahpixida]

Hi @vineetasharma105 @gannebamm - I have the radiation DAG ready, please can you review the linked PR and see if this implementation fits the use-case we discussed over the past couple of weeks.

Also will it alright if we saved the output as parquet files, or do you have something in mind on what the clipped output will look like

I'm still running more extensive tests on this feature - but so far the latitude/longitude inputs I've used for the testing hasn't been returning any clipped outputs, will it be ok if you could share some inputs on your end that you feel will definitely produce some results for the radiation geo-data

To give you a run-through again of the clipping workflow, it goes like this:

• We first download the Geo-Data, then we use the user inputs to generate a buffer-extent (using the code snippet you provided a couple of days ago @vineetasharma105 )
• Since the buffer-extent is defined by the minimum and maximum coordinates in both the horizontal (X) and vertical (Y) dimensions and is represented as (minX, minY, maxX, maxY) - we convert this coordinates to the min-max lat/long positions
• We then use this converted extents to clip the points on the downloaded netCDF4 data
• Finally this will be saved as a parquet file

[brightemahpixida]

I just added a new commit to the PR :)