zarr-developers / VirtualiZarr

Create virtual Zarr stores from archival data files using xarray syntax
https://virtualizarr.readthedocs.io/en/stable/api.html
Apache License 2.0
123 stars 24 forks source link

Store test datasets in this repo #226

Open TomNicholas opened 3 months ago

TomNicholas commented 3 months ago

Our current approach to testing involves a bunch of fixtures which each download a tutorial dataset from xarray (and cache it because it uses pooch), saves them to a temporary directory, then open that dataset from disk. This is not ideal for a few reasons:

  1. The datasets aren't minimal, so they contain more complexity than is really needed to test a single bug / feature. This can make debugging more complicated.
  2. We're using the network when we don't need to be.
  3. vz.open_virtual_dataset calls xr.open_dataset, but because of our test setup xr.open_dataset can potentially be called more than once in the same test invocation, even if the code we are testing only calls it once. This again can make debugging more confusing than it needs to be.

We do need to test our ability to read files from disk, but it might be better just to make some really tiny netCDF files and save them in this repo.