Request for amendment to BUFR Table D, to create a new BUFR template suitable for the distribution of XCTD data

[fbringas-noaa]

Initial request

This is a request to create a new BUFR template that is suitable for the distribution of XCTD data. Currently, template TM315004 is intended for XBT and XCTD temperature profile data. However this template does not include descriptors for the salinity profiles collected using XCTD probes. A new template including descriptors for salinity values would allow the complete distribution of the observations collected by XCTDs and could also be used for XBT data.

Amendment details

Currently, template TM315004 is used to submit XBT and XCTD temperature profiles to the GTS. However in the case of XCTD data, only the temperature can be submitted, as there are no descriptors in this template for salinity. A possible alternative is the use of a different template for XCTD data (for example TM315007), which allows for the submission of both temperature and salinity data, but this or other alternatives do not include a complete list of recommended metadata for XCTD probes.

The amendment requested is for a new template that can accommodate all the data and metadata required for distribution of XCTD observations. This new template can be created as a modification of TM315004, with the inclusion of 3 descriptors at the end of this template to account for salinity values from XCTD probes.

The document in the attachment provides details on the proposed new template. The proposed new template only includes 3 new descriptors (compared to TM315004) for salinity at the end of the template, which are highlighted in green in the document. All other lines in this proposal remain the same as in TM315004.

XCTD_BUFR_Template_proposal.pdf

3 15 006		XCTD temperature and salinity profile data	Note
001079	Unique Ship or mobile land station identifier for the profile	Hexadecimal string
001011	Ship or mobile land station identifier	Ship's call sign
001103	IMO Number. Unique Lloyd's register	= 0 to 9999999
001087	WMO marine observing platform extended identifier	(see Note 1)
001019	Long station or site name	Ship name
001080	Ship line number according to SOOP
005036	Ship transect number according to SOOP	(see Note 2)
001036	Agency in charge of operating the observing platform
001013	Speed of motion of moving observing platform
001012	Direction of motion of moving observing platform
301011	Year, month, day
301012	Hour, minute
301021	Latitude/longitude (high accuracy)
007032	Height of sensor above local ground (or deck of marine platform)
007033	Height of sensor above water surface
002002	Type of instrumentation for wind measurement
011002	Wind speed
011001	Wind direction
007032	Height of sensor above local ground (or deck of marine platform)
007033	Height of sensor above water surface
012101	Temperature/air temperature
012103	Dewpoint temperature
007032	Height of sensor above local ground (or deck of marine platform)	Set to missing (cancel)
007033	Height of sensor above water surface	Set to missing (cancel)
302021	Waves
002031	Duration and time of current measurement
002030	Method of current measurement
022005	Direction of sea-surface current
022032	Speed of sea-surface current
022063	Total water depth
008080	Qualifier for GTSPP quality flag
033050	Global GTSPP quality flag
022178	XBT/XCTD launcher type
022177	Height of XBT/XCTD launcher	Above sea level 0 to 50 m in units of whole m
022067	Instrument type for water temperature/salinity profile measurement
008041	Data significance
026021	Year
026022	Month
026023	Day
022068	Water temperature profile recorder types
025061	Software identification and version number
008041	Data significance	Set to missing (cancel)
008080	Qualifier for GTSPP quality flag	Set to missing (cancel)
002171	Instrument serial number for water temperature profile measurement
302090	Sea/water temperature high precision
002171	Instrument serial number for water temperature profile measurement
002032	Indicator for digitization	(see Note 3)
109000	Delayed replication of 9 descriptors
031002	Extended delayed descriptor replication factor
007063	Depth below sea/water surface (cm)
008080	Qualifier for GTSPP quality flag	= 13 Water depth at a level
033050	Global GTSPP quality flag
022043	Sea/water temperature
008080	Qualifier for GTSPP quality flag	= 11 Water temperature at a level
033050	Global GTSPP quality flag
022064	Salinity
008080	Qualifier for GTSPP quality flag	= 12 Salinity at a level
033050	Global GTSPP quality flag

Comments

The changes requested cannot be done in the sequence TM315004 itself as it would make encoding and decoding BUFR bulletins dependent on the version of the tables used. A new template based on adding new descriptors to TM315004 would allow for the submission of complete data and metadata for XCTDs, as well as XBT, without any ambiguity as the probe type can be specified with the corresponding descriptor.

Requestor(s)

Francis Bringas, National Oceanic and Atmospheric Administration (NOAA), co-chair of the Ship Of Opportunity Programme Implementation Panel (SOOPIP) of the Ship Observations Team (SOT)

Stakeholder(s)

This amendment will impact institutions submitting both XBT and XCTD data into the GTS, and weather forecast and modeling centers receiving data through the GTS for operational applications

Publication(s)

Manual on Codes (WMO-No. 306), Volume I.2, BUFR Table D, table reference 3 15 004 (XBT temperature profile data sequence)

Expected impact of change

MEDIUM

Collaborators

Joaquin Trinanes, NOAA, USA Rebecca Cowley, CSIRO, Australia Lisa Krummel, BOM, Australia

References

No response

Validation

No response

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.21.22.November.2023 notes: team looked at the proposal briefly and agrees to the proposal; Marijana can update the branch; Ask proposers for some samples;

[amilan17]

@fbringas-noaa can you provide some sample data within a week or two?

[fbringas-noaa]

The following zip archive provide examples of XCTD original data and the resulting BUFR bulletin using the proposed sequence.

XCTD_BUFR_proposal_examples.zip

The files in this zip archive are:

1. _XCTDfull.EDF : This is an example of the original XCTD data. This format is not unique or standard; it depends on the software used to collect the data. In this example some of the metadata appears in the first 35 lines, and the profile data in lines 50 and below. The parameters that would be submitted to the GTS are in the Data section (line 50 and below) in columns 3 (Depth), 4 (Temperature), and 6 (Salinity). Additional metadata included in the proposed BUFR template but not included in this file can be added to the BUFR bulletin by a different configuration file.
2. _XCTDfull.BUFR : This is an example of the data and metadata from the previous file, encoded in BUFR following the sequence proposed here.
3. _XCTDfull.DEC : This is the result of decoding the previous file in BUFR into an easy to read ASCII file. Notice that in Section 3, 'Data descriptors unexpanded' actually contain the expanded sequence of descriptors proposed in this template; this should be replaced by the new unexpanded descriptor when approved.
4. _XCTDfull.GTS : This is an example of a final bulletin in BUFR, ready for the GTS (including the GTS header), encoded following the sequence proposed here.
5. _XCTD_sample.EDF, XCTD_sample.BUFR, XCTD_sample.DEC, XCTDsample.GTS : These files are analogous to files in 1. through 4. above, respectively. In these examples only the first 25 lines of data in the original XCTD profile were used, for simplicity. Typical XCTD profiles may contain several thousand lines of Depth, Temperature, and Salinity records.

[marijanacrepulja]

@amilan17 I have updated the branch @fbringas-noaa could you please generate samples using sequence number instead of a list of descriptors. I have chosen first available number which is 315006. Could you also set the master table to 42. I apologise for any inconvenience. This is to satisfy the validation procedure. Many thanks

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.21.29.November.2023 notes:

in progress; Marijana reviewed the samples and provided feedback (see comment above);

[fbringas-noaa]

@marijanacrepulja : The examples provided previously were encoded/decoded using the tables from ECMWF here, where the latest version available is v37 (bufrdc_tables_wmo_v37.tar.gz). Do you know where can I download v42 of these tables?

For the second request: I am not sure how to generate a sample using unexpanded descriptor 315006 when that sequence is not defined in the tables. In this case the encoder will exit with an error as it cannot verify that the message is following a previously define sequence. For this reason in the previous examples I used the full list of descriptors in order to be able to generate a sample.

[marijanacrepulja]

Hi @fbringas-noaa,

Would you be able to create D0000000000000042000.TXT that contains list of the descriptors in your proposal under sequence 315006 . It is similar to already existing 315004. In your program master table should be set to 42.

You may need to also create B0000000000000042000.TXT and C0000000000000042000.TXT, those could be a copy of B0000000000000037000.TXT and C0000000000000037000.TXT

To be on safe side, it would be better to copy tables in some test directory and set the path to the tables. Version 42 will be released in May next year, and we dont have it. Your proposed BUFR template will be part of version 42. For time being to be able too create samples and validated them we need to mimic as the tables are version 42.

Hope this helps and let me know if more info is needed. Thanks Marijana

[fbringas-noaa]

Hi @marijanacrepulja

Following your instructions, in the attachment below there are 3 examples of actual XCTD profile data encoded in BUFR with Data descriptors unexpanded: 315006 and Master table version number: 42 (files _XCTD1.*; _XCTD2.*; _XCTD3.*).

XCTD_BUFR_examples_315006_v42.zip

The files in this zip archive are:

1. *_.EDF_**: Original XCTD data recorded in-situ.
2. *_.BUFR_**: Example of previous EDF files encoded in BUFR
3. *_.DEC_**: Examples of previous BUFR files decoded into ASCII
4. *_.GTS_**: Example of previous BUFR files ready to submit to the GTS (including the GTS header)
5. BUFR tables B, C, and D used for these examples are also included. As instructed, tables B, C are a copy of those for v37, while table D is that for v37 but modified to include the proposed sequence 315006.

Thank you so much for your help.

Francis

[marijanacrepulja]

XCTD.ecmwf.decoded.zip

Hi @fbringas-noaa,

Many thanks for providing samples. I was able to decode BUFR files provided using ecCodes. Also I did cross check values in the files, all look good.

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.15.December.2023 note:

@amilan17 validate the branch

[amilan17]

@marijanacrepulja or @fbringas-noaa Can you please update the issue summary to include a table of the new template?

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.10.January.2024 notes:

@marijanacrepulja will update the issue proposal; @amilan17 to validate the branch;

[marijanacrepulja]

@amilan17 I have updated proposal and included the new template, so you can validate it with the branch. Thanks