GTS bulletins compilation with WSI and TSI messages

[efucile]

Compilation of a GTS bulletin including reports of several stations requires the consideration of how to proceed if the stations are producing a mix of data with and without WSI.

[efucile]

According to Manual on GTS and Manual on Codes

• only one BUFR message per GTS message
• BUFR has one subset per reporting station
• each BUFR subset has the same descriptors and therefore stations in different subsets must be either with or without WSI

I see only the following options at the moment

1. add WSI to messages with only TSI and set WSI to missing value.
2. define a new T1 in the TTAAii GTS heading to separate TSI and WSI messages.

[efucile]

Option 2 is safer as would allow users who are not able to process WSIs to filter them out at GTS header level and would avoid the addition of artificial WSI elements to messages that are already in use. Option 2 is manageable as a GTS level configuration and should not require major software changes. The required change will be the ability of MSS software to make BUFR collectives with WSIs. Option 2 issue is that we have very few T1 letters free and we need one for CF-NetCDF as well.

T1	Data type	T2	A1	A2	ii	Priority
A	Analyses	B1	C1	C1	**	3
B	Addressed message	***	***	***	***	1/2/4*
C	Climatic data	B1	C1	C1	**	4
D	Grid point information (GRID)	B2	C3	C4	D2	3
E	Satellite imagery	B5	C1	C1	**	3
F	Forecasts	B1	C1	C1	**	3
G	Grid point information (GRID)	B2	C3	C4	D2	3
H	Grid point information (GRIB)	B2	C3	C4	D2	3
I	Observational data (Binary coded) – BUFR	B3	C6	C3	**	2
J	Forecast information (Binary coded) – BUFR	B3	C6	C4	D2	3
K	CREX	B3	C7	C3	**	2
L	Aviation information in XML	B7	C1	C1	**	1/2/3
M	–
N	Notices	B1	C1	C1	**	4
O	Oceanographic information (GRIB)	B4	C3	C4	D1	3
P	Pictorial information (Binary coded)	B6	C3	C4	D2	3
Q	Pictorial information regional (Binary coded)	B6	C3	C5	D2	3
R	–
S	Surface data	B1	C1/C2	C1/C2	**	2/4*
T	Satellite data	B1	C3	C4	**	2
U	Upper­air data	B1	C1/C2	C1/C2	**	2
V	National data	(1)	C1	C1	**	(2)
W	Warnings	B1	C1	C1	**	1
X	Common Alert Protocol (CAP) messages
Y	GRIB regional use	B2	C3	C5	D2	3
Z	–

@wmo-im/tt-wsi please comment on this

[efucile]

I would like to propose the following modification to the T1 table

T1	Data type	T2	A1	A2	ii	Priority
M	Observational data with WIGOS Station Identifier (Binary coded) – BUFR	B3	C6	C3	**	2

@wmo-im/tt-tdcf, @wmo-im/tt-wsi and @wmo-im/tt-protocols please examine this proposal and make comments by 27/11/2020

[SimonElliottEUM]

I support the addition of T1 = M as proposed. I suggest to add a little something extra to make clear that T1 = I should not be used if WIGOS Station Identifier is included. As the proposal stands, T1 = I or M would both be acceptable when WSI is present and this is not good.

[efucile]

I support the addition of T1 = M as proposed. I suggest to add a little something extra to make clear that T1 = I should not be used if WIGOS Station Identifier is included. As the proposal stands, T1 = I or M would both be acceptable when WSI is present and this is not good.

Thanks, @SimonElliottEUM here is the amended proposal as per your excellent suggestion

T1	Data type	T2	A1	A2	ii	Priority
I	Observational data without WIGOS Station Identifier (Binary coded) – BUFR	B3	C6	C3	**	2
M	Observational data with WIGOS Station Identifier (Binary coded) – BUFR	B3	C6	C3	**	2

[kaiwirt]

First of all a question: Do we really need another T1 for this? In my understanding the only purpose is to not mix WSI and TSI? In principle this could be left open to the centers (for instance IS////01 and IS////02 would also work without adding something to the headers)

Apart from that, "M" would work for T1. But i am still not happy with it for several reasons

1: We already distribute Israeli BUFR data with Wigos Station Identifiers as ISND02 LLBD on the GTS 2: How about High-Res Upper Air data (IUKD32 EDZW for instance also already uses WSI)? Do we need to change all this? 3: Isn't this in conflict with WIS2.0 where we want to phase out TTAAii in the next few years? 4: Centers not able to read WSI can just skip these messages.

5. It might be difficult to have two different data Dissemination streams (I and M) for data from the same synoptic network.

Additionally i wonder if there a reason why not take the chance to cut out some dead wood. WSI observations are something new, so centers need to update their systems anyway in order to use this data. So we could also say WSI will only be available on the WIS and not assigned new TTAAii headers. This of course needs to consider that this can only be done for new stations or new data types. Having two seperate data Dissemination streams (be it files or bulletins or I and M) for the same data has its pitfalls.

At one point someone has to say TTAAii is over and start to use WIS. Why not take this step now. Freeze TTAAii. And put new data out without headers.

[jbathegit]

I understand how this might be useful, but we need to be aware that this change would likely have a significant impact on GISCs and RTHs and their internal processes and tables, not to mention all users of such products who would now have to change all of their own processes to accept ^M as well as ^I for numerous observational data types. Plus, as @efucile mentioned, all producers of such messages would need to adapt their encoding processes as well.

I also note that this would be the first instance of a letter in the T1 table which differs from another solely due to the presence or absence of a particular data value, and not in the actual category (e.g. Forecast, Warning, Surface, Upperair) or overall format (e.g. BUFR, GRIB, TAC, XML) of the data. So this seems a bit of a stretch in that regard, not to mention the removal of one of the few remaining free letters, as @efucile also noted.

Again, I understand how this might be useful from an overall routing perspective to allow centers to easily switch on and off their receipt of such products. But the reality is that all users are going to have to deal with a mix of products (those with WSI and those without) at some point anyway, since this won't be an immediate, coincident transition for all producers, and thus there will still be a significant overlap period when centers need to be able to seamlessly process both types.

So for these reasons my vote would be for option 1.

[marianmajan-ibl]

I supports Jeff's opinion that having a new GTS T1 header for WSI data would have significant impacts on NMCs. And I would be strongly against Kai's proposal to issue observations with WSI in bulletins without GTS headers.

I may be missing the basic point here – what is a problem of parallel dissemination of observations with WSI / TSI with the same T1T2A1A2 differentiated by ii part of the heading only? Currently, there is already quite a lot of centres that started dissemination of reports with WSI in ^I bulletins (e.g. ISA///). What problems does it cause for processing centres?

Also, e.g. ISA/// is used for bulletins with reports with n-minute AWS data (TM 3 07 092) as well as hourly AWS data (TM 3 07 091). Or, ISM, ISI, ISN may contain reports with TM 3 07 080, TM 3 07 086, 3 07 096 or 3 07 079 (and others, e.g. national BUFR sequences such as ISND38 AMDS produced by DWD). If the reception centre is capable of processing TM 3 07 080 reports, it does not have to be capable of processing TM 3 07 079. Similarly centre may be able to process TM 3 07 080 but can not process TM 3 07 080 preceded by TM 3 01 050. I don't see any differences of parallel WSI/TSI dissemination comparing to the current dissemination of various BUFR templates in bulletins with the same T1T2A1A2 part of GTS header.

[JMauror]

Hi. I agree with Marian´s comments. I think if we define some ii range is much better them having another T1 letter. If WIS 2.0 comes along, probably the abbreviated heading will be discontinued.

[efucile]

@wmo-im/tt-wsi @wmo-im/tt-tdcf @wmo-im/tt-protocols Thank you all for your comments. I will try to answer all of the comments and I see that we need to refine our requirements to get to an effective solution to this. I prefer to start from the requirements and I would suggest everyone to contribute as you think appropriate. We need to have a complete discussion on this.

[KenRJTD]

I'm not sure, but if we really need to recognize BUFR message with-WSI or without-WSI in the GTS (before decoding), I think “ii” is better than T1=”M”. However, please note that the range of “ii” for some BUFRs are limited and little bit complicated. For example. IUS 01–19 TEMP (parts A, B, C, D) IUS 20–39 TEMP MOBIL (parts A, B, C, D) IUS 40–59 TEMP SHIP (parts A, B, C, D)

[just for your reference] There may be another option in the Manual on the GTS, utilizing file exchange according to the Attachment II-15 with General File Naming Convention (GFNC). This option will have positive and negative impacts, but we may be able to use same TTAAiiCCCCYYGGggBBB and to recognize the message, if it has WSI or TSI before decoding by [_freeformat].

GFNC: pflag_productidentifier_oflag_originator_yyyyMMddhhmmss[_freeformat].type[.compression] For example: BUFR with TSI: A_ISIC20RJTD280900CCA_C_RJTD_20201128092917_TSI.bin BUFR with WSI: A_ISIC20RJTD280900CCA_C_RJTD_20201128092917_WSI.bin

Also SYNOP and CREX can use GFNC. For example: SYNOP with TSI: A_SIJP20RJTD280900CCA_C_RJTD_20201128092917_TSI.txt CREX with TSI: A_KSIC20RJTD280900CCA_C_RJTD_20201128092917_TSI.txt

[imallas]

Countries Reporting WSI on GTS

wsi.pptx

[imallas]

List of Stations reporting WSI WSI_cases (1).xlsx

[kaiwirt]

When checking the list of @imallas we noticed, that several bulletins are sent out with the same TTAAii but with different templates. That is the same TTAAii is received twice, one time with WSI and one time without.