ua-snap / rasdaman-ingest

Collection of ingredients/configurations + docs for ingesting data into Rasdaman
MIT License
3 stars 0 forks source link

Update to tas_2km_projected Rasdaman ingest #78

Open BobTorgerson opened 5 months ago

BobTorgerson commented 5 months ago

We want to mirror the table design in the current version of the ArcticEDS for precipitation that shows the minimum, mean, and maximum across all 5 models and 3 scenarios in the temperature section:

Precipitation image

Temperature image

Currently, we don't have all 5 models or all 3 scenarios available in the tas_2km_projected Rasdaman coverage, and only have 2 models: GFDL-3M3 & NCAR-CCSM4 (along with the 5ModelAvg) and 2 scenarios: RCP 4.5 & RCP 8.5. To allow for us to get the minimum, mean, and maximum across all of the models and scenarios, we will need to re-ingest the files for all of that data.

As I recall, we ended up going against this because of how long it takes to generate the temperature data for the ArcticEDS when we had the full dataset. However, with the newest version of Rasdaman on Zeus and the prospect of getting Rasdaman Enterprise, there may be some serious performance gains to look into for this dataset.

If we were to go forward with ingesting this dataset with the full coverage, we would need to download all of the GeoTIFFs in this TAS 2km dataset for the monthly data. We would then need to run:

https://github.com/ua-snap/rasdaman-ingest/blob/tas_projected_2km/iem/tas_projected_2km/merge.py

After merging the TIFs into NC files for tas, tasmax, and tasmin for each model, we would again merge those various variables as bands within an overarching NC file that would contain all of the variables for a given model and month combination using this script:

https://github.com/ua-snap/rasdaman-ingest/blob/tas_projected_2km/iem/tas_projected_2km/combine.py

After that was finished, we would need to modify the ingest.json file in the following areas to accomadate the new models and scenarios: https://github.com/ua-snap/rasdaman-ingest/blob/9d6febc5670c799a299868d018bee7a43202fe57/iem/tas_projected_2km/ingest.json#L35C1-L38C1 https://github.com/ua-snap/rasdaman-ingest/blob/9d6febc5670c799a299868d018bee7a43202fe57/iem/tas_projected_2km/ingest.json#L39-L42 https://github.com/ua-snap/rasdaman-ingest/blob/9d6febc5670c799a299868d018bee7a43202fe57/iem/tas_projected_2km/ingest.json#L179

And modifying the luts.py file to include the correct number of scenarios.

This would get us halfway there for getting the temperature table the way we want, and the rest would require changes to the data-api /eds/temperature endpoint.