Abstract out ADF's time series generation functionality so other components can use it - Githubissues

NCAR / CUPiD

CUPiD is a “one stop shop” that enables and integrates timeseries file generation, data standardization, diagnostics, and metrics from all CESM components.

https://ncar.github.io/CUPiD/

Apache License 2.0

21 stars 19 forks source link

Abstract out ADF's time series generation functionality so other components can use it #30

Closed TeaganKing closed 3 months ago

TeaganKing commented 7 months ago

We should also be sure that this is not a duplicated effort (eg, with Brian Dobbin's tool).

This could also be an area where we could leverage GeoCAT.

dabail10 commented 6 months ago

The ADF tool just uses ncrcat and is not very flexible. The main things we want are chunking and compression. Should this be part of issue #18 ?

TeaganKing commented 5 months ago

In our discussion during the CUPiD hackathon, with @nusbaume @brianpm @justin-richling @kafitzgerald and myself, we decided we'll want to pull out adf_diag.create_time_series(); this will entail a slight rewriting. We want to avoid circular dependencies. Let's plan to make a timeseries.py file within CUPiD/cupid. Eventually, if we pull in Brian Dobbins' timeseries repository (if it's actually useful to replace), we can replace the timeseries.py file.

TeaganKing commented 5 months ago

We could merge the ADF taylor diagram with GeoCat Viz taylor diagram.

Taylor_plot_setup, plot_taylor_data, and taylor_plot_finalize could probably be replaced by geocat's function. We probably can't replace gathering data from ADF, selecting variables, subsetting for seasonal averaging, etc.

TeaganKing commented 5 months ago

Overarching CUPiD needs: timeseries, regridding, climatology/time averaging, website generation functionality.

Prioritization:

Website generation move into CUPiD -- This currently uses jinja and should be within CUPiD itself rather than universal outside of CUPiD.
GeoCAT could be really useful for regridding and ADF has more flexible time averaging (eg, 4 month seasonal averages or single).
Put timeseries into CUPiD -- if we are prioritizing this, it would be good to know which components need timeseries vs history files?
Determine if some of the more generalized plotting functions are more useful in GeoCAT viz or in CUPiD . (Within ADF/lib/plotting_functions.py, we may want to pull mask_land_or_ocean into a centralized place.) See plots described in #6

TeaganKing commented 5 months ago

Moving forward:

[x] Have GeoCAT member attend ADF hackathon on a Thursday afternoon or another meeting time -- determine which functions we can make into GeoCAT functions - @kafitzgerald can coordinate with GeoCAT to see who can join.
[ ] Making top level CUPiD website - @justin-richling and @TeaganKing can collaborate on this.
[x] See if timeseries is needed for other components during next CUPiD meeting @TeaganKing will add to agenda
[ ] @justin-richling - time averaging comparison with ADF and GeoCAT comp

TeaganKing commented 5 months ago

From a discussion with @mnlevy1981 regarding the timeseries note-- we'll eventually want to have all components be able to run on timeseries in order to use smaller model output files, but we should probably first focus on some of the other items from the ESDS event hackathon last week.