search

meggart / DiskArrays.jl

Other
79 stars 16 forks source link

Redesign of GridChunks type #47

Closed meggart closed 2 years ago

meggart commented 2 years ago

This PR implements a re-design of the GridChunks type which defines the chunk sizes for a DiskArray. The main purpose of this redesign is to provide more first-class support for arrays with non-constant chunk sizes. This happens quite frequently when concatenating NetCDFs along an axis that does not have constant length (for example concatenating annual NetCDF files along time axis where some have a leap year and some don't). An example use case is ConcatDiskArray from DiskArrayTools.jl.

This PR makes it possible to use views, reshape, broadcast and permutations also on these irregular arrays. Since we change the way that GridChunks works, I would bump the minor version here although that name is not exported. This would also give us a bit more time for consolidation and testing and dependents can decide when they want to update their deps sections and check if tests pass. @rafaqz it would be nice if you can test this with a recent DimensionalData and Rasters version as well

rafaqz commented 2 years ago

Thanks, sounds good. I will test with Rasters.jl in the next few days. And yes please do a minor version bump. GridChunks probably should be part if the formal interface even if not exported.

meggart commented 2 years ago

I have tested this update with several downstream packages and so far everything seems ok, so I would like to go ahead and merge, if there are no objections.

rafaqz commented 2 years ago

Sorry I've been on the road a bit. I've tested with Rasters.jl and had no problems.