search

roocs / dachar

BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

During scan, check consistency of variable attributes #31

Closed agstephens closed 4 years ago

agstephens commented 4 years ago

Description

As part of the scan process we need to bypass xarray and use raw netCDF4 to check the following:

  1. Variable with var_id exists in all files
  2. Main attributes (which we need to agree on) are the same across all files for that variable (var_id)
  3. Coordinate variables (except time) of the main variable are identical across files
    • including their attributes
    • and values
  4. Coordinate variable time should have:
    • consistent attributes
    • non-overlapping and contiguous time steps
    • get Ag to pull out time-checks code from the past.

If any of our consistency checks fail we raise an Exception and log that we could not perform a full scan due to issue SOME SENSIBLE STATEMENT ABOUT THE ERROR, and we move on.

Example code:

from netCDF4 import Dataset
ds = Dataset('/badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/rcp85/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_HadGEM2-ES_rcp85_r1i1p1_200512-203011.nc')
ds.variables()
ds.variables
ds.variables.keys()
var_dict = dict([(attr, getattr(ds.variables['tas'], attr)) for attr in ds.variables['tas'].ncattrs()])
var_dict
import readline
dir(readline)
readline.write_history_file('hist.py')