Compliance checkers to verify AcMeta and SONAR-netCDF4 conventions

[ghost]

@Haris-Kunnath commented on Apr 17, 2020, 1:24 AM UTC:

It would be useful if we could provide compliance checker(s) possibly written in Python, R, or Matlab to check compliance of NetCDF (as an example) with latest AcMeta and SONAR-netCDF4 convention documents.

From an implementation point of view, a user at the end of the day may want to check whether the data generated is compliant with these conventions. This would help foster data sharing and reuse in future.

Asking from AcMeta point of view, is there any way to automatically harvest all global attributes listed in the document and potentially store in a CSV or some other format? Or any trick to quickly read these attributes (and associated obligation fields) from the HTML or XML file.

We have an in-house Matlab function (with preconditions) written for checking AcMeta convention along with other metadata authorities – not elegant but did warn the user about missing global and variable attributes as you may see in the attached image.

Cheers

-Haris

This issue was moved by gavinmacaulay from ices-eg/wg_WGFAST#56.

[ghost]

@gavinmacaulay commented on Apr 17, 2020, 3:03 AM UTC:

I agree that a conformance checker would be useful and quite likely necessary. For ease of use, a online version might be best, but this could just be a front-end to something in python, R, etc.

There is cfchecker.py, which checks for CF compliance, which sonar-netcdf4 tries to conform to. But this isn't checking anything specific to acmeta or sonar-netcdf4.

To automatically harvest the acmeta attributes, I suggest turning the problem around - store the attribute tables as csv files(s) and then include them into the asciidoc file (asciidoctor will accept tables as delimiter-separated values). There might be some gotcha's to do with formatting of the tables for asciidoctor (see here), but hopefully not crucial.

[cyrilleponcelet]

Hi, For information I developed a description file of the model of the sonar netcdf format that I use to generate some python, java and even asciidoctor documentation. The asciidoctor part is compared with the one here to check that I forgot nothing, we could easily make some code allowing to check for mandatory attributes and so on. I could deliver everything to the community (maybe as a side repo), it's not the best code ever but it does the job

[cyrilleponcelet]

I created a prototype of Sonar-netcdf checker. It will open a netcdf file and allow to check if mandatory attributes and variables are defined. Recommanded attributes trigger a warning and right now nothing if done for Mandatory if Applicable (needs some specific coding) A check is also done on type definition.

The source code is located here : https://gitlab.ifremer.fr/fleet/formats/pySonar-netcdf , more specifically in the sonar_netcdf/sonar_validator.py and sonar_netcdf/sonar_validator_model.py

If you want to test it, you can click on the binder icon and start a binder notebook for interactive test, you will need to upload your dataset to the binder docker image (they are deleted afterward)

The Sonar-netcdf checker is avalaible through the notebook Sonar-netcdf Validator.ipynb, the model used is up to date the status at that time (and including #53)

Let us know what you think of this proposal.

[gavinmacaulay]

Nice! I'll have a closer look at it in the next few days.

There is also https://github.com/ioos/compliance-checker/, which can accommodate plugins to suit particular conventions (suggested by @emiliom).

[cyrilleponcelet]

Hi everyones, We update our sonar-nertcdf validator to be compliant with sonar-Netcdf4 2.0 model.

There is still work to do to have an full checker (more specifically for Mandatory if Applicable fields), but this tools is helpfull to check if all mandatory variables are declared.

The source code is located here : https://gitlab.ifremer.fr/fleet/formats/pySonar-netcdf , more specifically in the sonar_netcdf/sonar_validator.py and sonar_netcdf/sonar_validator_model.py

Next step for us is to package everything and deliver through pypi and have a look at the https://github.com/ioos/compliance-checker/ if we could add it as a plugins.

[gavinmacaulay]

Is there an option to choose checking for version 1 of the convention?

[cyrilleponcelet]

Unfortunately we started to work on it during the transition from 1 to 2 version Be do not have a description of the model that is available for the 1 version