wmo-im / BUFR4

BUFR edition 4
MIT License
27 stars 9 forks source link

non-CCITTIA5 characters in GitHub .csv file(?) #55

Closed jbathegit closed 3 years ago

jbathegit commented 3 years ago

Branch

https://github.com/wmo-im/BUFR4/tree/issue55
https://github.com/wmo-im/CCT/commits/bufrIssue55

Final Proposal

add when ready

Original Discussion Point

Hello @wmo-im/tt-tdcf

Maybe I'm mistaken, but it was my understanding that everything in the csv files should be part of the CCITT IA5 character set(?)
I'm asking because a colleague of mine found a couple of entries in https://raw.githubusercontent.com/wmo-im/BUFR4/master/BUFRCREX_CodeFlag_en_02.csv which aren't part of this character set; specifically, there are ‰ (parts-per-thousand) symbols in the meaning descriptions for code figures 1 and 2 of the code table for 0-02-033.

Is this permissible? One of the problems with having these sorts of things in csv files is that they require multiple bytes to represent them (for example, ‰ is equivalent to hexadecimal e280b0) , which in turn complicates automated string searches and other processing when using a standard keyboard. I realize this is a valid symbol in full UTF-8, but for some reason I thought we were sticking just to CCITT IA5 for the csv files that we distribute, so that every character can be represented within a single byte.

amilan17 commented 3 years ago

@DavidBerryNOC @jbathegit @SimonElliottEUM 

Please review https://github.com/wmo-im/CCT/commits/bufrIssue55

I moved all edits for CCT tables into this branch and removed the extra columns added for ia5.  As discussed during the meeting these edits do not change the meaning and therefore don't need to go through the FT amendment process. 

@jbathegit FYI,  - https://github.com/wmo-im/CCT/commits/55temp is a copy of https://github.com/wmo-im/CCT/commits/issue55 and I reset CCT/issue55 to represent the relevant CCT issue only.

amilan17 commented 3 years ago

we will introduce these mods in the upcoming FT release of the machine readable codes.

SimonElliottEUM commented 3 years ago

@amilan17 You asked us to review https://github.com/wmo-im/CCT/commits/bufrIssue55. but I am not really sure what I should be looking at. I assume the changes I made have all been retained.

amilan17 commented 3 years ago

This is the diff -- I just want confirmation that I didn't mess up anything when I cleaned up the branches. 

https://github.com/wmo-im/CCT/compare/bufrIssue55

jbathegit commented 3 years ago

The https://github.com/wmo-im/CCT/compare/bufrIssue55 branch looks OK to me, w.r.t. all of the CCTs.

@SimonElliottEUM, I believe that all of your and @SibylleK changes to the regular BUFR code/flag tables have been retained in https://github.com/wmo-im/BUFR4/tree/issue55 and are still set to go forward. @amilan17 please correct me if I'm wrong.

Thanks to all on this thread for your help with resolving this issue!!

SimonElliottEUM commented 3 years ago

@SimonElliottEUM, I believe that all of your and @SibylleK changes to the regular BUFR code/flag tables have been retained in https://github.com/wmo-im/BUFR4/tree/issue55 and are still set to go forward. @amilan17 please correct me if I'm wrong.

@jbathegit Thanks Jeff - and Happy Birthday! 🥳

efucile commented 3 years ago

Happy birthday @jbathegit

david-i-berry commented 3 years ago

@amilan17 all good from me.