Indices arrays are typically u64, even when they rarely need to be that wide. That means during decompression, we unpack into far more memory than necessary.
During write time, perhaps in the same operation that (doesn't exist yet) resets offsets to zero (i.e. shifts boolean arrays to be zero-aligned), and trims unused dictionary values.
Alternatively, we can use stats at read-time and have some sort of "canonicalize_smallest / canonicalize_into(DType)" function that lets us decompress into the smallest viable width.
Indices arrays are typically u64, even when they rarely need to be that wide. That means during decompression, we unpack into far more memory than necessary.
During write time, perhaps in the same operation that (doesn't exist yet) resets offsets to zero (i.e. shifts boolean arrays to be zero-aligned), and trims unused dictionary values.
Alternatively, we can use stats at read-time and have some sort of "canonicalize_smallest / canonicalize_into(DType)" function that lets us decompress into the smallest viable width.