Appropriate validation at different stages of dataset creation

[mhidas]

There are different levels of validation we can apply at different stages:

• Schema validation: making sure each individual dictionary (dimensions, variables and attributes) has items that makes sense for its purpose. This is what's currently done with jsonschema at template creation time. It can't be too restrictive because it may not make sense to fully specify all dataset elements in a JSON template (e.g. a generic template could specify the attributes of a TEMP variable, without saying what dimensions it should have).
• Schema completeness: Similar to validation, but this level should insist on having all the information required to create a valid netCDF4 object. This could be done using a more strict set of json schemas.
• Consistency: This is to ensure that the three dictionaries are consistent with each other; in particular, that the dimensions of variables (and their data arrays) are consistent with what is specified in the dimensions dictionary.

I think validation can be applied at the beginning, but checking for completeness and consistency can be left until just before creating the file. The code is doing something close to this already, but we need to make sure it's doing the right amount of validation at each stage.

[ocehugo]

I would rename it to match the usage:

1. Template validation [a part of a netcdf file: a writable/non-writtable structure]
2. Netcdf validation [a writtable structure]

[mhidas]

@ocehugo I'm not sure I understand your suggestion. How do the concepts of schema validation, completeness and consistency (as described above) map to your names?

I think these concepts are independent of whether you're talking about a netcdf file or a template. Actually the only difference is that a DatasetTemplate object is able to represent an incomplete, inconsistent template, but you can't write a netcdf until you've satisfied all three conditions.

(So maybe I've just answered my own question...)

[mhidas]

Just to get it clear in my head before I start messing around in the code, here's what I'm thinking

At template creation time, just validate the dictionaries (or JSON) provided against the schema we have already specified. Checks include

• All attribute and variable names are valid;
• Only allowed (but at this stage not required) special keys are present (e.g. "_dimensions", "_datatype");
• All specified attributes have valid types.

At netCDF writing time (just before attempting to write the file)

1. Validate the schema again, as above;
2. Ensure that all the necessary special keys are present, automatically filling in missing information if possible (e.g. set "_datatype" attribute from type of the array assgined to "_data").
3. Ensure consistency
   • between "_dimensions" of variables and those defined for the dataset (auto update their sizes from the data arrays if not set in template); and
   • between "_datatype" and type of "_data" for each variable.