Faster chunk checking for backend datasets

pydata / xarray

N-D labeled arrays and datasets in Python

https://xarray.dev

Apache License 2.0

3.63k stars 1.09k forks source link

Open dcherian opened 9 hours ago

dcherian commented 9 hours ago

[x] Closes #8902 (shaves 30s off the runtime, dask is responsible for the rest)
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst

dcherian commented 6 hours ago

I didn't re-think through the logic but the cache idea makes sense, thanks!

The core of it didn't change, just avoided materializing a huge iterable in memory in favor of finding the first disagreement.