search

wmo-im / GRIB2

GRIB2
MIT License
22 stars 9 forks source link

Validate Machine Readable Codes #60

Closed amilan17 closed 3 years ago

amilan17 commented 3 years ago

Please validate the XML and TEXT files for GRIB. GRIB2_r26_candidate.zip

Thanks!

erget commented 3 years ago

@amilan17 it looks like there's some problems here.

When importing CodeFlag.txt and Template.txt my software imported 150 and 129 elements correctly, respectively.

The XML did not fare as well - CodeFlag.xml imports 90 elements with 3 errors. These are in these elements:

  1. 4.225
  2. 4.230
  3. 4.233

In each case "codefigure" had no value and thus I couldn't import it.

I wasn't able to import the templates as XML. I think this is a bug on our side and am investigating that currently.

amilan17 commented 3 years ago

odd. Can you email me the list of elements that successfully imported? that might help me narrow down the problem.... The three elements have very little information, but they remain unchanged from version 25....

erget commented 3 years ago

I'm waiting to be able to deploy a bugfix, let me try it out with the updated version first - the problem might be on our side.

erget commented 3 years ago

@amilan17 I've now deployed the bugfix and could import 129 elements from Templates.xml. No errors, so from an XML formatting perspective it looks like our software can work work with it.

@SibylleK had mentioned that she had some quality concerns about the new files - I've not had time to look into their content, so my review should be understood as restricted to the correct formatting of the tables.

sebvi commented 3 years ago

We are looking at the csv/txt files.

One thing we have noticed that is not related to the csv format, but the content is that table 1.0 has NOT been updated with the latest versions of the table. Since this one is version 26, we should have entries for version 24 with the date it became operational version 25 with the date it became operational version 26 with 16th November I guess.

and the next line should be updated to be "27-254 Future versions"

it seems we have differences in the format compared to the usual csv/txt files. stay tuned for details

shahramn commented 3 years ago

Both CSV files lack the "No" column that used to be there in previous versions (this was the first column which had the row numbers in it) Previous columns were: "No","Title_en","SubTitle_en","CodeFlag","Value","MeaningParameterDescription_en","Note_en","UnitComments_en","Status"

Now we have: Title_en,SubTitle_en,CodeFlag,Value,MeaningParameterDescription_en,Note_en,UnitComments_en,Status

amilan17 commented 3 years ago

@sebvi It appears that this table has not been maintained as it should. I will update, starting on line https://github.com/wmo-im/GRIB2/blob/feature_script/GRIB2_CodeFlag_1_0_CodeTable_en.csv#L25.

amilan17 commented 3 years ago

@shahramn - We removed the "No" from the output, because they are not unique identifiers and will change every time there are additions. Is it critical for your software?

shahramn commented 3 years ago

@shahramn - We removed the "No" from the output, because they are not unique identifiers and will change every time there are additions. Is it critical for your software?

No. It is not critical at all. We can change our scripts but other centres may hit this problem if their scripts expect 9 instead of 8 columns.

amilan17 commented 3 years ago

@shahramn Do you think it's reasonable to ask/expect other centres to adjust the their copy of the output (add the the "No" column with arbitrary values) or modify their scripts (to not expect this column)?

shahramn commented 3 years ago

@shahramn Do you think it's reasonable to ask/expect other centres to adjust the their copy of the output (add the the "No" column with arbitrary values) or modify their scripts (to not expect this column)?

Well. It should not be my decision of course (wink). But in my humble opinion if the WMO make an announcement stating that there is a change in the format of the CSV file and also give a good reason for it, then other users can make the necessary adjustments in their scripts.

What is Enrico's opinion about this?

sebvi commented 3 years ago

@amilan17: by design, code Table 1.0 is maintained by the Secretariat. I suggest that you update the table at the time you create the branch with all the merge from the proposal. If you don't know the date you could leave the date open until you know. But it is very important to have this table up to date.

I think a similar table exists for the CCT, maybe it is woirth checking if that one is up to date as well.

amilan17 commented 3 years ago

Well. It should not be my decision of course (wink). But in my humble opinion if the WMO make an announcement stating that there is a change in the format of the CSV file and also give a good reason for it, then other users can make the necessary adjustments in their scripts.

What is Enrico's opinion about this?

Enrico doesn't want the "No" values included and I believe there was a decision captured in an issue in the BUFR repo. An official announcement appears necessary.

jbathegit commented 3 years ago

@amilan17 @efucile

Just to clarify, once the XML tables are finalized for BUFR v35 and GRIB v26, will they be posted at https://community.wmo.int/activity-areas/wmo-codes/manual-codes/latest-version? Or will we have to clone them directly from GitHub? Or will they be posted at a different URL?

Just curious what the official protocol will be going forward, and when we should expect these latest updates to be formally released - thanks! ;-)

amilan17 commented 3 years ago

We are looking at the csv/txt files.

One thing we have noticed that is not related to the csv format, but the content is that table 1.0 has NOT been updated with the latest versions of the table. Since this one is version 26, we should have entries for version 24 with the date it became operational version 25 with the date it became operational version 26 with 16th November I guess.

and the next line should be updated to be "27-254 Future versions"

@sebvi -It appears that maintenance of this table stopped, because it's deprecated.

"Note: This code table is deprecated. See Common Code table C–0 instead." - p. 139 of the manual

amilan17 commented 3 years ago

Just to clarify, once the XML tables are finalized for BUFR v35 and GRIB v26, will they be posted at https://community.wmo.int/activity-areas/wmo-codes/manual-codes/latest-version? Or will we have to clone them directly from GitHub? Or will they be posted at a different URL?

@jbathegit - I will post these to https://community.wmo.int/activity-areas/wmo-codes/manual-codes/latest-version, but we will have conversations in the TT about changing this in the future.

amilan17 commented 3 years ago

@SibylleK had mentioned that she had some quality concerns about the new files - I've not had time to look into their content, so my review should be understood as restricted to the correct formatting of the tables.

@SibylleK - have you had a chance to test these GRIB files?

sebvi commented 3 years ago

on a side note, indirectly related to generating files for the release, we use to have a "Human readable" pdf produced, containing a diff with the modification since the previous version. And a final pdf with the full tables updated (and the modification highlighted in red)

any plans to create these?

tomkralidis commented 3 years ago

+1 @sebvi. I would also advocate for outputting these to a webpage (as we started doing in our GRIB3 setup awhile back).

erget commented 3 years ago

Yes, or at least having the CSVs and human-readable documents stored in a repo so that you can diff between versions.

amilan17 commented 3 years ago

@tomkralidis @sebvi @shahramn @erget I created a new issue for this discussion. Please provide feedback there. Thanks! https://github.com/wmo-im/CCT/issues/29

amilan17 commented 3 years ago

Machine readable codes are published to https://community.wmo.int/activity-areas/wmo-codes/manual-codes/latest-version