Decide on a method to compare Zarr in object store with original archive netCDF

cedadev / cmip6-object-store

CMIP6 Object Store Library

BSD 3-Clause "New" or "Revised" License

4 stars 4 forks source link

Open agstephens opened 4 years ago

agstephens commented 4 years ago

Suggested approach:

For each dataset:

[x] map ID to archive dir
[x] select one NC file
[x] if bigger than threshold then continue (THRESHOLD=250MB)
[x] read dataset 1 (from NC file)
[x] read dataset 2 (from Zarr)
[x] compare datasets by value, variables, coordinates, attributes and variable attributes
[x] log result
[x] if processed N files, then exit

agstephens commented 4 years ago

Implemented as a test:

$ python tests/cmip6_zarr/test_compare_zarr_nc.py

We should move this to:

$ python cmip6_object_store/cmip6_zarr/cli.py verify -p cmip6 [--dataset-ids <dataset_id>,<dataset_id>]