search

intake / intake-esm

An intake plugin for parsing an Earth System Model (ESM) catalog and loading assets into xarray datasets.
https://intake-esm.readthedocs.io
Apache License 2.0
130 stars 42 forks source link

Use `chunks="auto"` by default in `xarray_open_kwargs` #633

Closed dougiesquire closed 8 months ago

dougiesquire commented 8 months ago

Change Summary

Very small change to default to chunks="auto" rather than chunks={} when opening files with xarray. This is motivated by recent changes in xarray to the behaviour of chunks={}.

Related issue number

Closes #632

Checklist

dougiesquire commented 8 months ago

Ah, unfortunately auto rechunking does not work with object dtype. This is probably too common for it to make sense to use chunks="auto" by default. Some options:

Thoughts @mgrover1, @dcherian?

mgrover1 commented 8 months ago

Ah, unfortunately auto rechunking does not work with object dtype. This is probably too common for it to make sense to use chunks="auto" by default. Some options:

  • try to use chunks="auto" and revert to chunks={} if that fails;
  • use chunks=-1, which gives the same behaviour as pre-v2023.09.0 chunks={} (prior to v2023.09.0, these did the same thing);
  • ditch this and stick with chunks={}

Thoughts @mgrover1, @dcherian?

I think sticking with chunks={} by default makes the most sense here still... any thoughts here @dcherian or @andersy005 ?

andersy005 commented 8 months ago

Ah, unfortunately auto rechunking does not work with object dtype. This is probably too common for it to make sense to use chunks="auto" by default. Some options:

  • try to use chunks="auto" and revert to chunks={} if that fails;
  • use chunks=-1, which gives the same behaviour as pre-v2023.09.0 chunks={} (prior to v2023.09.0, these did the same thing);
  • ditch this and stick with chunks={}

Thoughts @mgrover1, @dcherian?

I think sticking with chunks={} by default makes the most sense here still... any thoughts here @dcherian or @andersy005 ?

i vote in favor of sticking with chunks={} unless implementing the work around above ( * try to use chunks="auto" and revert to chunks={} if that fails;) doesn't result in complicated code :(

dougiesquire commented 8 months ago

Thanks @mgrover1 and @andersy005. I agree with you both so I'll close this PR