xarray-contrib / xarray-simlab

Xarray extension and framework for computer model simulations
http://xarray-simlab.readthedocs.io
BSD 3-Clause "New" or "Revised" License
73 stars 9 forks source link

Handle output snapshots for dynamic sized arrays #109

Closed benbovy closed 4 years ago

benbovy commented 4 years ago

We need to handle dynamically sized arrays when taking multiple snapshots during a simulation.

There are different ways to solve this:

  1. concatenate snapshots along the dynamically sized dimension(s): see #108
  2. maybe resize the output (zarr) array when writing new data, so that the final shape corresponds the the max size for each dynamically sized dimension: see #111
  3. create a new output variable for each snapshot (e.g, process__varname__0, process__varname__1, etc.).

Each of these solutions has its own advantages and limitations. We can implement each of them.

benbovy commented 4 years ago

Looks like option 1 could be easily derived from option 2 by chaining stack and dropna on the output Dataset, e.g.,

out_ds.stack(midx_dim=('clock', 'dim')).dropna('midx_dim')

Since #102, option 2 wouldn't be that bad regarding storage (depending on chunk size vs. contiguous regions of missing values), so #108 is probably not worth after all.

benbovy commented 4 years ago

Closed in #111