kraemer-lab / DART-Pipeline

Data analysis pipeline for the Dengue Advanced Readiness Tools (DART) project
https://dart-pipeline.readthedocs.io
MIT License
2 stars 0 forks source link

Add scripts to process meteorological Data - APHRODITE #10

Open rowannicholls opened 11 months ago

rowannicholls commented 11 months ago

Now that we're downloading the following data:

We need to add the ability to process these (mostly nc.gz) files. @Winter-23 mentioned that he has scripts to do this, so it would be good to integrate these into the existing pipeline.

rowannicholls commented 11 months ago

FYI @jsbrittain @prathyushsambaturu

prathyushsambaturu commented 11 months ago

Additional notes: 1999-2022 (time frame of interest) for daily mean temperature. How ever we could download all data from 1960 as it in the scripts now.

Methods for aggregation of temperature and precipitin data suggested by Iago: statisticall downscaling, inverse distance weighted interpolation, etc.

Winter-23 commented 11 months ago

When we are talking about aggregating we are talking to go from low resolution to district level right? (From 25 km to 2 I mean)

prathyushsambaturu commented 11 months ago

We need go from (i) 25km to district level (admin2) and (ii) 25km to ward level (admin3). I am assuming, whether these transitions will be lower/higher resolution depends on the size of the corresponding admin region.

Winter-23 commented 10 months ago

I would suggest regridding the data from the 25 km to level admin 2 and 3. This is, we use an interpolation method, such as the inverse distance weighted interpolation, to artificially increase the resolution of the dataset. I think it might be plausible to do it using CDO (Climate Data Operators), which can be used within in a Python environment. Nonetheless, we would have to consider 2 things:

1.- The obtained dataset will weight much more than the original, so I do not recommend using it in the raw data, we should first load the dataset, then select the area of study, and then apply the interpolation. (All of this can be done using CDO)

2.- Because the values obtained come from an interpolation, they might deviate from the observed in the surface, but as we commented at the Hackton, we could use them to test the pipeline.

Also, the nc.gz files are compressed nc files (or netcdf files), so maybe the course of the script would be something similar to:

1.- Decompress the file

2.- Select the area

3.- Perform the interpolation.

Would it be useful for you if I code a Python script to do this?

abhidg commented 5 months ago

@rowannicholls @Winter-23 following up on this - is this implemented?

abhidg commented 2 months ago

@rowannicholls is this implemented in the current version?