VirtualiZarr

[oruebel]

@rly @magland I just came across VirtualiZarr, which looks related. The effort looks like it somewhat grew out of kerchunk but tries to provide similar functionality in a more Zarr-native form.

• https://github.com/zarr-developers/VirtualiZarr
• https://virtualizarr.readthedocs.io/en/latest/

The Pangeo / climate modeling community seems to also look at this: https://github.com/esgf2-us/esgf-virtual-zarr-data-access

[magland]

Thanks @oruebel this does seem relevant. I started looking at those links but haven't really grasped out it works yet.

[oruebel]

Another possibly interesting thing maybe https://docs.hdfgroup.org/hdf5/v1_14/_h5_f__u_g.html#title11 In particular, the family, split, and multi driver allow splitting data across multiple files under the hood.