The existing ncdata.netcdf4.from_nc4 takes a very crude approach to dask,
so far only using dask "auto" chunking
~so far not even using "auto" chunking but insisting on the variable fitting in a single chunk~
~-- that at least could surely be improved on ??~
The rationale is that we generally expect to take our data loads from iris/xarray.
But the "secondary usage" for file adjustments is still a valid future development.
We should probably allow for an intelligent chunking strategy and/or direct (per-variable) user control.
pp-mo
commented
8 months ago
The existing
ncdata.netcdf4.from_nc4takes a very crude approach to dask, so far only using dask "auto" chunking ~so far not even using "auto" chunking but insisting on the variable fitting in a single chunk~ ~-- that at least could surely be improved on ??~The rationale is that we generally expect to take our data loads from iris/xarray.
But the "secondary usage" for file adjustments is still a valid future development. We should probably allow for an intelligent chunking strategy and/or direct (per-variable) user control.