Are you guys aware of Kerchunk? It creates a virtual Zarr Store that maps to individual chunks in a bunch of netCDF files. This blog post explains it better than I can.
It might be very useful for BOUT++ data as you can run kerchunk on your thousands of netCDF files once, then from that point on treat the whole lot as if it were one single Zarr store, and you then don't have to do the slow xarray.open_mfdataset step each time you open your data.
Are you guys aware of Kerchunk? It creates a virtual Zarr Store that maps to individual chunks in a bunch of netCDF files. This blog post explains it better than I can.
It might be very useful for BOUT++ data as you can run kerchunk on your thousands of netCDF files once, then from that point on treat the whole lot as if it were one single Zarr store, and you then don't have to do the slow
xarray.open_mfdatasetstep each time you open your data.