Closed scottyhq closed 2 years ago
Mind moving this to #253 ?
Mind moving this to https://github.com/corteva/rioxarray/issues/253 ?
🤦 sorry i overlooked that! Should I just copy/paste and close this one?
sorry i overlooked that! Should I just copy/paste and close this one?
No worries. I overlook things all the time :smile:. Yes, please.
It could be nice to take advantage of known raster tiling (rasterio
.block_shapes
) to assign dask chunks when opening tiled datasets. It seems straightforward to implement with xarray's backend preferred chunk sizes.Setting this could also avoid common errors with exploding task graphs when users rely on defaults (chunks=True) or reading way more data than necessary (chunks={}).
According to xarray docs for open_dataset/open_dataarray, "chunks={} loads the dataset with dask using engine preferred chunks if exposed by the backend, otherwise with a single chunk for all arrays. chunks='auto' will use dask auto chunking". Below is an example with some public data (18287x18460 with 256x256 tiling):
There are many considerations for optimal chunks (see dask blog), but a good default size could be determined by ~100Mb and a multiple of 256x256?