HDFGroup / hsds

Cloud-native, service based access to HDF data
https://www.hdfgroup.org/solutions/hdf-kita/
Apache License 2.0
126 stars 52 forks source link

Use numcodecs VLenArray for encoding variable length dataset chunks #263

Open jreadey opened 9 months ago

jreadey commented 9 months ago

Doc on VLenArray is here: https://numcodecs.readthedocs.io/en/stable/vlen.html#vlenarray. I'm pretty sure that this will be more performant than the run length encoding that is done in hsds/util/arraydata.py currently. Care will be needed to ensure that data written in the original schema is still readable.