Thomas-Moore-Creative
commented
4 months ago Keeping in mind these helpful suggestions from @dcherian here: https://github.com/Thomas-Moore-Creative/Climatology-generator-demo/issues/16#issuecomment-2080144828
/// quote:@dcherian ///
Some general comments (applicable to latest flox/xarray)
- If you are not grouping by multiple variables, you can use the much nicer
ds.groupby("time.month").mean()syntax and it will use flox automatically if installed. - Over the past few months, I've worked on setting
methodautomatically looking at how the groups are distributed across chunks. You should not have to setmethod. This will let flox choose what's appropriate to whatever chunking you have. - That said (2) runs on heuristics so it's possible that it doesn't make the right choice. Details will help, particularly what are you grouping by, and how is the big array chunked along the core dimensions of the groupby operation.
- Are you running in to trouble with the groupby reduction, or on calculating anomalies w.r.t the grouped-reduced values? With dask, you want to do those two steps separately, otherwise memory issues will often result.
- It's tempting to simply say
.chunk(time=30)for e.g. but really you should think about rechunking to a frequency (e.g. monthly or two-monthly). We don't have nice syntax for this yet (https://github.com/pydata/xarray/issues/7559) but you can quite easily figure out the appropriate chunk tuple withds.time.resample(time="2M").count(). If properly done, flox will then choose either "cohorts" or "blockwise" automatically for you, and save some memory. Here's an example: https://flox.readthedocs.io/en/latest/user-stories/climatology.html#rechunking-data-for-cohorts
dcherian
Title: Generating descriptive statistics composited over ENSO phase for very large ocean reanalysis data using xarray, dask, and flox.
float32data over nearly 9000 netcdf file assets in total. For this problem we’ll focus on only the daily 3Dtempvariable.GadiGadisupercomputer ( national science resources - not public ) using a custom beta catalog I wrote usingintake-esmandecgtools- https://github.com/Thomas-Moore-Creative/BRAN2020-intake-catalogtempvariable stored in ashortdata type with native chunking like:temp:_ChunkSizes = 1, 1, 300, 300netcdffile has one month of daily data and so the length ofTimevaries from 28 to 31 for any one of the 366 files concatenated together. The compressed 5GB singlenetcdffile becomes 34GB in loadedfloat32tempvariable xarray object is `Time: 11138 st_ocean: 51 yt_ocean: 1500 xt_ocean: 3600and **11.16TB** infloat32`xarray_open_kwargsare employed to increase that, however I believe thatTimechunks on lazy load can’t be larger than the size of each individualnetcdffile?xarray_open_kwargsofTime:31we’ll get actual chunks like: [31,28,31,30,31,30 . . . ] and some years will be [ …. 31,29,31,30 . . .]medianandquantile) as both monthly and daily climatologies from daily data organised as 366 x one-month-per-netcdf file.xarray_open_kwargsdo we use on the initial loading.open_mfdataset?ChunkSizesgroupbyoperations viaflox?zarrcollection be useful. If so whatTimechunking to choose? all-time ({’Time’:-1}or a monthly frequency?floxmethods are best to choose?