Since for raster data often times some of the data represents a coordinate, some 2D or 3D chunks can have a lot of redundancy in them, along the second (and third) dimension. For example for a 2D array where we keep track of the latitude and the longitude, for a given chunk either variable will only actually vary along one of the 2 dimensions.
Since those variables will often times be included in filter conditions, and since we currently support skipping all the data to be read if no values of variables in the filter match the condition(s), for a given chunk of a zarr store, reading those "coordinate" variables as 1D arrays and broadcasting them as 2D or 3D arrays in code might give some nice performance improvements, since I/O is often times pretty expensive.
To specify that a particular variable is to be broadcasted after it's read, we will simply use the attributes in the metadata.
Since for raster data often times some of the data represents a coordinate, some 2D or 3D chunks can have a lot of redundancy in them, along the second (and third) dimension. For example for a 2D array where we keep track of the latitude and the longitude, for a given chunk either variable will only actually vary along one of the 2 dimensions.
Since those variables will often times be included in filter conditions, and since we currently support skipping all the data to be read if no values of variables in the filter match the condition(s), for a given chunk of a zarr store, reading those "coordinate" variables as 1D arrays and broadcasting them as 2D or 3D arrays in code might give some nice performance improvements, since I/O is often times pretty expensive.
To specify that a particular variable is to be broadcasted after it's read, we will simply use the attributes in the metadata.