Ensure that there are mechanisms for local validation of WMO Codes Registry contents

[braeckel]

At the ICAO AsiaPac IWXXM Workshop it was indicated that having local copies is an essential operational function, and when IWXXM goes fully operational with IWXXM 2.1 and particularly into IWXXM 3 in 2020 it is very important that data producers and consumers be able to validate WMO Codes Registry contents locally. There are thousands of producers and consumers who cannot reasonably make dynamic requests to http://codes.wmo.int whenever they receive a message.

There are two potential solutions: 1) The WMO Codes Registry has a full download function for all IWXXM codes or the entire registry contents 2) Add Schematron checks into IWXXM (probably somewhat incomplete) to ensure that a URL starts with "http://codes.wmo.int/306/4678", for example

[blchoy]

I have a suggestion back in 2013 with regard to (2): https://wiswiki.wmo.int/tiki-download_file.php?fileId=527

[braeckel]

In TT-AvXML-6 it was agreed that this is an important capability. It was also agreed that downloading a local copy (#1) is the proper solution.

It was agreed that there should be a script for updating local code registry contents and that this script should be part of the IWXXM release so it can be used to update the codes between IWXXM releases. It was also agreed that we would provide local copies of the code list entries relevant to IWXXM with each IWXXM release.

[braeckel]

A new bin/codeListsToSchematron.py file was checked in for this capability for translating RDF codelists to Schematron rules. It searches through IWXXM schemas for vocabulary metadata (under appinfo), downloads the corresponding codelist RDF file from codes.wmo.int, and generates a codelists.sch file which uses the RDF files to verify that the appropriate xlink:href's are correctly referenced.

Before this issue can be closed it should be determined if and how the resulting codelist Schematron and RDF files should be merged with the existing process for generating Schematron. Is this a single iwxxm.sch file generated by the release process, or a separate .sch file? If this is merged with the existing Schematron generation process there may be changes required to the new script for integrating with the existing scripts

[braeckel]

Also note that this script does not include support for checking O&M types against the appropriate IWXXM code lists due to the fact that this information is not in the IWXXM schemas. This should be considered for IWXXM 3, depending on the results of the simplification task (#27)

[braeckel]

In the TT-AvXML telecon today it was agreed that a new script should be added to merge the results of the bin/codeListsToSchematron.py file into the existing Schematron definition. Therefore only a single Schematron file should be used with IWXXM.

[braeckel]

Reopened as it was agreed that for 2.1.1RC1 we will add an extra, optional iwxxm-collect-codelists.sch file that includes these rules. In 3.0RC1 the codelists will be part of the release (i.e., part of iwxxm.sch, see #60) but introducing this change to a bugfix is perhaps not appropriate. This allows for evaluation and testing in 2.1.1RC1 but will not break any existing 2.1.0 files.

@marqh will create this file and close this issue once WMO Codes Registry changes are complete and iwxxm-collect-codelists.sch is created.

[blchoy]

This is closed together with Pull Request #64.