Open TomNicholas opened 3 months ago
Passing chunks=None to open_mfdataset should return lazily-indexed numpy arrays, like open_dataset does.
Can't do this without virtual concat machinery (https://github.com/pydata/xarray/issues/4628) which someone decided to implement elsewhere 🙄 ;)
We could change the default to chunks={}
in anticipation though.
Can't do this without virtual concat machinery (https://github.com/pydata/xarray/issues/4628) which someone decided to implement elsewhere 🙄 ;)
😅
It's still broken at the moment though - I had a (ridiculous) case where I don't care that concat will load everything in memory, I just want to completely avoid creating dask.array
objects, and right now there is no possible input option to open_mfdataset
to do that.
We could change the default to chunks={} in anticipation though.
That's probably more useful, as well as actually being consistent.
See #5704 for changing chunks={}
and more discussion.
What happened?
Passing
chunks=None
toxr.open_dataset
/open_mfdataset
is supposed to avoid using dask at all, returning lazily-indexed numpy arrays even if dask is installed. Howeverchunks=None
doesn't currently work forxr.open_mfdataset
as it gets silently coerced internally tochunks={}
, which creates dask chunks aligned with the on-disk files.Offending line of code: https://github.com/pydata/xarray/blob/12123be8608f3a90c6a6d8a1cdc4845126492364/xarray/backends/api.py#L1040
What did you expect to happen?
Passing
chunks=None
toopen_mfdataset
should return lazily-indexed numpy arrays, likeopen_dataset
does.Minimal Complete Verifiable Example
MVCE confirmation
Relevant log output
Anything else we need to know?
As the default is
None
, changing this without changing the default would be a breaking change. But the current behaviour is also not intended.Environment
main