Validate Machine Readable Codes - v27 Release Candidate

[amilan17]

RC

https://github.com/wmo-im/GRIB2/releases/tag/v27RC.4

Summary and purpose

To ensure that the machine-readable codes v27RC are valid and ready for testing during the comment period. 

Action proposed

1. [x] Update script to ensure consistent output order
2. [x] identify key individuals to test
3. [x] Confirm file validation

Discussions

Machine-Readable Files

• [x] CodeFlag.txt
• [x] Template.txt
• [x] CodeFlag.xml
• [x] Template.xml

[SimonElliottEUM]

Code/Flag looks strange.

XML is problematic because of None types being used. We've had this before. Must be analysed by inspection of XML file to figure out what's going on, probably omitted tags. The elements in question do this:

<GRIB2_CodeFlag_en>
<Title_en>Code table 4.225 - Weather</Title_en>
<CodeFlag/>
<MeaningParameterDescription_en>(see FM 94 BUFR/FM 95 CREX Code table 0 20 003 - Present weather)</ MeaningParameterDescription_en>
<Status>Operational</Status>
</GRIB2_CodeFlag_en>

The "naked" tag is what's causing the problems - this was also the case in the last one. Vocabulary Manager expects this to be expressed as an empty double-tag, i.e. <CodeFlag></CodeFlag>

[SimonElliottEUM]

The CodeFlag XML file looks like it contains 94 elements total, CSV has 151. Please explain

[amilan17]

@erget @SimonElliottEUM -- 

The CodeFlag XML file looks like it contains 94 elements total, CSV has 151.

I'm unable to find a count of 151 "elements". 

The source code has 94 CodeFlag tables represented in CSV files. 

When I evaluate the machine-readable codes, I see that the TXT file has 2,248 lines (including header) and the XML file has 2,247 \<GRIB2_CodeFlag_en> nodes. In a spreadsheet analysis I see that both have 94 CodeFlag tables.

[erget]

@amilan17 here are the imported GRIB2 Code/Flag table entries. I got them out of the CSV.

I've inspected all of these manually and it looks legitimate to me - all entries are code/flag tables, there are 151 of them, and they are all unique as far as I can tell (I was checking by reading through, don't have a computer programme for it). My guess is that the XML file is missing entries?

If you get a chance, please have a look at the linked site above and let me know if I'm missing something.

[amilan17]

@erget - the difference is how the EUMETSAT Vocabulary Manager handles Code table 4.2.  

Code table 4.2 - Parameter number by product discipline and parameter category == is one table in the source CSV, XML and TXT on GitHub. e.g. 

Title_en	SubTitle_en	CodeFlag	Value	MeaningParameterDescription_en
Code table 4.2 - Parameter number by product discipline and parameter category	Product discipline 0 - Meteorological products, parameter category 0: temperature	0		Temperature
Code table 4.2 - Parameter number by product discipline and parameter category	Product discipline 0 - Meteorological products, parameter category 0: temperature	1		Virtual temperature

However, the vocabulary manager lumps all the values into one entry for each unique combination of title, title_en, ~CodeFlag and meaning.~ Thus multiplying the number of entries. 

_id	name	values/0/codeentry	values/0/codefigure
4.2.0.0	Code table 4.2 - Parameter number by product discipline and parameter category # Product discipline 0 - Meteorological products, parameter category 0: temperature	Temperature	0
4.2.0.1	Code table 4.2 - Parameter number by product discipline and parameter category # Product discipline 0 - Meteorological products, parameter category 1: moisture	Specific humidity	0

[amilan17]

@erget - on a different note, you said "I got them out of the CSV". Is the vocabulary manager importing the source *.csv files or the  *.txt files in the csv folder?

[erget]

@amilan17 good eye, that's the rub. I imported the 2 files in the CSV folder: CodeFlag.txt and Template.txt.

[amilan17]

validation complete