search

ocean-transport / pleiades_llc_recipes

Examples of analysis of MITgcm LLC simulations of NASA Pleiades supercomputer
MIT License
8 stars 2 forks source link

Metrics to test chunking on OSN #12

Open cspencerjones opened 1 year ago

cspencerjones commented 1 year ago

I have started a list of possible metrics over on hackmd

@shfer suggested looking at previous work on LLC4320 to think of other metrics and I think that's a good idea, but we also agreed that we probably don't want to spend a very long time on this.

cc @dhruvbalwada @rsaim @TomNicholas @rabernat

TomNicholas commented 1 year ago

Should we add what the "core dimensions" of each operation would be? i.e. the dimensions along which you ideally would not want the data to be chunked when performing this type of analysis.

For example if you wanted to compute the FFT of a timeseries to find temporal modes you ideally would not want it chunked along the time dimension, as the xrft package can't handle chunked core dimensions right now.

I added examples for the ones I know about.

cspencerjones commented 1 year ago

It seems as though we definitely want to test different chunksizes and compression levels.

@rabernat are you happy to stick with zlib for compression? The other kinds of compression do not seem to be implemented in xarray really.

rabernat commented 1 year ago

:+1: to only zlib