search

zarr-developers / VirtualiZarr

Create virtual Zarr stores from archival data files using xarray syntax
https://virtualizarr.readthedocs.io/en/stable/api.html
Apache License 2.0
120 stars 22 forks source link

Arbitrary chunking of uncompressed files (e.g. netCDF3) #86

Open TomNicholas opened 6 months ago

TomNicholas commented 6 months ago

@rabernat made the interesting point to me that uncompressed files (e.g. netCDF3 files) have no specific chunking, as you can start reading bytes from any point in the file immediately, with no minimum unit of decompression to do first.

I'm not totally sure what this implies for VirtualiZarr generating references from netCDF3 files, as it's still meaningful to talk about one chunk per file concatenated together in a manifest.

Perhaps this is something that should ultimately be taken account of in zarr readers: that reading bytes from an uncompressed array does not require loading an entire chunk into memory first.

TomNicholas commented 3 months ago

It seems great minds think alike https://medium.com/pangeo/using-kerchunk-with-uncompressed-netcdf-64-bit-offset-files-cloud-optimized-access-to-hycom-ocean-9008ba6d0d67

That post uses kerchunk.utils.subchunk (which I did not know existed), but I wonder if a neater API to use for a similar feature in virtualizarr could just be to add a chunks argument to open_virtual_dataset, that can only be used for uncompressed files?

TomNicholas commented 3 months ago

In fact one could imagine a .rechunk method on a ManifestArray, but it would raise unless the ManifestArray pointed to contiguous-on-disk uncompressed data.