search

aodn / imos-toolbox

Graphical tool for QC'ing and NetCDF'ing oceanographic datasets
GNU General Public License v3.0
46 stars 31 forks source link

schema validation for most matlab types, closes #577 #578

Closed ocehugo closed 5 years ago

ocehugo commented 5 years ago

This PR includes a schema validation utility.

It's included here as the primary tool for me (and hopefully others) to develop the toolbox with further confidence and provide enhanced checking when comparing and ensuring the data structures are what they should be.

This PR is self-contained and do not depend on anything related to the toolbox - only Matlab standard routines.

ocehugo commented 5 years ago

-I assumed you checked performance of this. I note that there are a few for loops and this can be problematic(some of them might be vectorised).

No, my calls and tests are fast enough that I didn't even bother looking into it. Calls with very deep nested structures usually take less than 10ms. The entire 102 tests take 7s in my machine (<10ms per test), including the overhead of the test suite (setup, defs, loads,etc).

Vectorization is not very applicable - I'm not dealing with fixed-sized arrays but with unknown nested levels of structs/cells in recursive mode (tree walk with unknown deep levels). Modern matlab also got a good jit compiler and for loops are sometimes faster than vectorized versions.

ocehugo commented 5 years ago

I just finished adding some missing help strings.

ocehugo commented 5 years ago

When using this toolbox today, I found two corner cases: comparisons with infs and nans were not working since they are considered doubles. Better to differentiate then, since @isinf and @isnan exists.

I also forgot to include the joinCell function used in the test setup. This is part of CellUtils. the commits above fix this

ocehugo commented 5 years ago

@ggalibert , all docstrings/licenses in.