wmo-im / BUFR4

BUFR edition 4
MIT License
27 stars 9 forks source link

New Table D sequence for reporting observations from ocean gliders #16

Closed david-i-berry closed 1 year ago

david-i-berry commented 4 years ago

Authors

Summary and purpose

The Ocean Gliders Data Management Task Team (OGDMTT) proposes a new BUFR sequence to report the full suite of glider observations along a trajectory profile.

Action proposed

The team is kindly asked to review and approve the contents for inclusion within the next update to the WMO Manual on Codes.

Discussions

Ocean glider data is currently submitted to the GTS using the Argo BUFR sequence (3-15-003) but this is not ideal. For example, the Argo sequence does not allow the along track positions to be reported as part of the trajectory profile and a number of parameters are missing. This new sequence seeks to address those deficiencies.

Detailed proposal (updated 10/01/2023)

New Table B Entries

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-42-016 Sea water potential density referenced to sea surface kg m-3 5 -10000000 24

New Table D Entries

Category 15: Oceanographic report sequences new sequence for Ocean glider trajectory profile

Table Reference Table References Element Name Description
3-15-012 Sequence for reporting observations from a single glider trajectory profile
3-01-150 (WIGOS identifier)
2-01-129 Change data width Increase width of following elements by 1 bit
0-01-087 WMO marine observing platform extended identifier
2-01-000 Change data width Cancel increase in width
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
0-22-032 Speed of sea surface current
0-22-005 Direction of sea-surface current
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
0-08-021 Time significance Set to 2, time averaged
0-04-025 Time period or displacement Duration of dive
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
1-26-000 Delayed replication of 26 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-23-000 Delayed replication of 23 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

See also https://github.com/wmo-im/CCT/issues/122. ## Test data [test-data.zip](https://github.com/wmo-im/BUFR4/files/10383572/test-data.zip)
kbailey-noaa commented 1 year ago

@vturpin et al, I agree with accelerating the validation and revisiting those parameters at a later time. Thanks!

castelao commented 1 year ago

@vturpin , I suggest using estimated chl instead of the fluorometer counts. If the provider do not convert it, I think it would be hard for anyone else do any good use of the raw counts without a lot of auxiliary information.

@david-i-berry , what do you need to keep Chl-a (0-41-006) in this batch? The sensors used in Spray could go from 0-150, with a nominal resolution up to 0.033. The usual operations actually run with higher gain. Think 0-15 / 0.33. [micro-grams/liter].

The sensor mentioned by @vturpin has a sensitivity of 0.025, measuring 0-50 according to Sea-Bird manual. [micro-grams/liter]

10e-6 is to convert the units to kg/m3. So a scale of 8 and reduce the width to 14 would have a better resolution than both sensors and save 3 bits per measurement while still covering up to ~160 microg/l. Please, double check it.

For testing values, one could use a 'profile' such as: 0, 0.01, 49.99, 50.00 [1e6 kg m-3]. Since the current format had a replicator per variable, it could be only these four measurements independent of the length of the other variables, is that right?

It might be worth considering a reference of -1e to help identifying bad calibrations that might otherwise pass by.

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference-12-and-13.12.2022 notes:

need sample data and validation before January in order to have something available for operational use in May 2023

castelao commented 1 year ago

@david-i-berry , what is your preferred way to send the sample data for chl to you?

vturpin commented 1 year ago

With regard to the status of this issue and the importance of moving on, I would rather not include CHLA in the validation process if it is not ready yet. We are about to miss again the train for the realease and that is not acceptable for the glider community.

The main point is: Are T and S ready to be operational in May 2023? If yes it should become operational in May 2023. Whether or the other variables are ready. If not, what is missing to achieve that.

However, can you clarify the following point too :

... | validated? 10 | Water pressure at a level 11 | Water temperature at a level 12 | Salinity at a level 13 | Water depth at a level 14 | Sea/water current speed at a level 15 | Sea/water current direction at a level 16 | Dissolved oxygen at a level 25 | Water conductivity at a level 26 | Sea water potential density referenced to sea surface at a level

... | pending ? 25 | Water conductivity at a level 26 | Sea water potential density referenced to sea surface at a level 27 | Mass concentration of chlorophyll A at a level 28 | Particle beam attenuation coefficient at a level 29 | Photosynthetically active radiation (400 - 700 nm) photon flux at a level 30 | Turbidity at a level 31 | Water pH at a level

@castelao, with regard to chla, I agree with you that only estimated chla should be transmitted.

amilan17 commented 1 year ago

@vturpin -- the goal is to have these proposals validated and ready for submission through the fast-track cycle by mid-January. I suggested before January, because this potentially gives us an opportunity to revalidate if changes are needed. However, there is roughly a two week window of wiggle room if needed. 

@castelao if the sample data is not too large you can drag and drop into a comment box of this issue.

david-i-berry commented 1 year ago

There are two issues:

1) validation of the new elements to make sure we have the scale, offset and reference values correct 2) validation of the sequence as a whole to make sure it makes sense after encoding and subsequent decoding.

The two parameters that were most problematic were PAR and backscatter, see https://github.com/wmo-im/BUFR4/issues/16#issuecomment-863866131. If we can drop these two this would remove one of the blocks for validation.

For ChlA, I am getting confused by the discussion. At the moment we have "Mass concentration of chlorophyll A in sea water" in kg m-3 and not the raw counts (unless this is the backscattering coefficient).

vturpin commented 1 year ago

@david-i-berry Thanks, Yes, drop PAR and Backscatter from the process. For CHLA, I think the confusion comes from a comment I made earlier. Sorry for that. It has been clarified that the count is not what is expected. The "Mass concentration of chlorophyll A in sea water in kg m-3" is what should be send.

castelao commented 1 year ago

One question, is it correct the sequence 0-02-169, 0-11-002, 0-11-001, 0-02-169 ? Is it needed 0-02-169 twice?

castelao commented 1 year ago

@vturpin , to speed up while keeping it clear, could you show here what would be the reduced table D entry (3-15-012) that you're thinking about?

vturpin commented 1 year ago

@castelao I have difficulty to understand the sequence so I am pretty sure I put element out of the reduced table D that should remain in. Note also this is my personal opinion, BGC variables out of the reduced table D, as I think RT transmission of those variables is not mature enough in the glider community. Also speed of current could be removed as glider transmit average current and not "current at a level" ** mean are off the reduced table D

Table Reference | Table References | Element Name | Description -- | -- | -- | -- 3-15-012 |   | Ocean glider trajectory profile sequence | Sequence for reporting observations from a single glider profile   | 3-01-150 | (WIGOS identifier) |     | 2-01-129 | Change data width | Increase width of following elements by 1 bit   | 0-01-087 | WMO marine observing platform extended identifier |     | 2-01-000 | Change data width | Cancel increase in width   | 0-01-019 | Long station or site name |     | 0-01-036 | Agency in charge of operating the observing platform |     | 0-02-148 | Data collection and/or location system |     | 0-01-085 | Observing platform manufacturer's model |     | 0-01-086 | Observing platform manufacturer's serial number |     | 0-08-021 | Time significance | Set to 25, nominal reporting time   | 3-01-011 | (Year, month, day) |     | 3-01-013 | (Hour, minute, second) |     | 3-01-021 | (Latitude / longitude (high accuracy)) |     | 0-11-104 | True heading of aircraft, ship or other mobile platform |    ** | 0-02-169 | Anemometer type | Set to 2, wind observation through ambient noise (WOTAN)**  ** | 0-11-002 | Wind speed |  ** ** | 0-11-001 | Wind direction |  **  ** | 0-02-169 | Anemometer type | Set to missing / cancel previous value **   | 0-22-032 | Speed of sea surface current |     | 0-22-005 | Direction of sea-surface current |     | 0-08-021 | Time significance | Set to 2, time averaged   | 0-04-025 | Time period or displacement | Duration of dive   | 3-01-011 | (Year, month, day) |     | 3-01-013 | (Hour, minute, second) |     | 3-01-021 | (Latitude / longitude (high accuracy)) |     | 0-22-031 | Speed of current |     | 0-22-004 | Direction of current |     | 0-08-021 | Time significance | Set to missing / cancel previous value   | 0-05-068 | Profile number |     | 0-01-079 | Unique identifier for profile |     | 1-26-000 | Delayed replication of 26 descriptors |     | 0-31-001 | Delayed descriptor replication factor | Number of profile sections (e.g. descending, horizontal, ascending)   | 0-22-056 | Direction of profile | 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)   | 1-23-000 | Delayed replication of 23 descriptors |     | 0-31-002 | Extended delayed descriptor replication factor |     | 3-01-011 | Year, month, day |     | 3-01-013 | Hour, minute, second |     | 3-01-021 | Latitude / Longitude (high accuracy) |     | 0-08-080 | Qualifier for GTSPP quality flag | 20, position   | 0-33-050 | Global GTSPP quality flag |     | 0-07-062 | Depth below water surface |     | 0-08-080 | Qualifier for GTSPP quality flag | 13, water depth at a level   | 0-33-050 | Global GTSPP quality flag |     | 0-22-065 | Water pressure |     | 0-08-080 | Qualifier for GTSPP quality flag | 10, water pressure at a level   | 0-33-050 | Global GTSPP quality flag |     | 0-22-045 | Sea/water temperature |     | 0-08-080 | Qualifier for GTSPP quality flag | 11, water temperature at a level   | 0-33-050 | Global GTSPP quality flag |     | 0-22-066 | Water conductivity |     | 0-08-080 | Qualifier for GTSPP quality flag | 25, water conductivity at a level   | 0-33-050 | Global GTSPP quality flag |     | 0-22-064 | Salinity |     | 0-08-080 | Qualifier for GTSPP quality flag | 12, salinity at a level   | 0-33-050 | Global GTSPP quality flag |     | 0-42-016 | Sea water potential density referenced to sea surface |     | 0-08-080 | Qualifier for GTSPP quality flag | 26, sea water potential density at a level   | 0-33-050 | Global GTSPP quality flag |     | 1-14-000 | Delayed replication of 14 descriptors |     | 0-31-001 | Delayed descriptor replication factor | Number of profile sections (e.g. descending, horizontal, ascending)   | 0-22-056 | Direction of profile | 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)   | 1-11-000 | Delayed replication of 11 descriptors |     | 0-31-002 | Extended delayed descriptor replication factor |   ** | 3-01-011 | Year, month, day |   ** | 3-01-013 | Hour, minute, second |   ** | 3-01-021 | Latitude / Longitude (high accuracy) |   ** | 0-08-080 | Qualifier for GTSPP quality flag | 20, position ** | 0-33-050 | Global GTSPP quality flag |   ** | 0-22-031 | Speed of current |   ** | 0-08-080 | Qualifier for GTSPP quality flag | 14, sea / water current speed at a level ** | 0-33-050 | Global GTSPP quality flag |   ** | 0-22-004 | Direction of current |   ** | 0-08-080 | Qualifier for GTSPP quality flag | 15, sea / water current direction at a level ** | 0-33-050 | Global GTSPP quality flag |   ** | 1-13-000 | Delayed replication of 13 descriptors |   ** | 0-31-001 | Delayed descriptor replication factor | Number of profile sections (e.g. descending, horizontal, ascending) ** | 0-22-056 | Direction of profile | 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal) ** | 1-10-000 | Delayed replication of 10 descriptors |   ** | 0-31-002 | Extended delayed descriptor replication factor |   ** | 3-01-011 | Year, month, day |   ** | 3-01-013 | Hour, minute, second |   ** | 3-01-021 | Latitude / Longitude (high accuracy) |   ** | 0-08-080 | Qualifier for GTSPP quality flag | 20, position ** | 0-33-050 | Global GTSPP quality flag |   ** | 2-01-130 | Change data width | Add 2 bits to data width ** | 0-22-188 | Dissolved oxygen |   ** | 2-01-000 | Change data width | cancel operation ** | 0-08-080 | Qualifier for GTSPP quality flag | 16, dissolved oxygen at a level ** | 0-33-050 | Global GTSPP quality flag |   ** | 1-18-000 | Delayed replication of 18 descriptors |   ** | 0-31-001 | Delayed descriptor replication factor | Number of profile sections (e.g. descending, horizontal, ascending) ** | 0-22-056 | Direction of profile | 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal) ** | 1-15-000 | Delayed replication of 15 descriptors |   ** | 0-31-002 | Extended delayed descriptor replication factor |   ** | 3-01-011 | Year, month, day |  ** ** | 3-01-013 | Hour, minute, second |  ** **   | 3-01-021 | Latitude / Longitude (high accuracy) |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 20, position **   | 0-33-050 | Global GTSPP quality flag |   **   | 0-41-006 | Mass concentration of chlorophyll A in sea water |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 27, mass concentration of chlorophyll A at a level **   | 0-33-050 | Global GTSPP quality flag |   **   | 1-04-000 | Delayed replication of 4 descriptors |   **   | 0-31-001 | Delayed descriptor replication factor |   **   | 0-02-071 | Spectrographic wavelength |   **   | 0-41-007 | Volume backwards scattering coefficient of radiative flux in sea water (BBP) |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 28, particle beam attenuation coefficient at a level **   | 0-33-050 | Global GTSPP quality flag |   **   | 0-02-071 | Spectrographic wavelength | cancel **   | 1-11-000 | Delayed replication of 11 descriptors |   **   | 0-31-001 | Delayed descriptor replication factor | Number of profile sections (e.g. descending, horizontal, ascending) **   | 0-22-056 | Direction of profile | 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal) **   | 1-08-000 | Delayed replication of 8 descriptors |   **   | 0-31-002 | Extended delayed descriptor replication factor |   **   | 3-01-011 | Year, month, day |   **   | 3-01-013 | Hour, minute, second |   **   | 3-01-021 | Latitude / Longitude (high accuracy) |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 20, position **   | 0-33-050 | Global GTSPP quality flag |   **   | 0-41-008 | Photosynthetically active radiation (400-700 nm) photon flux |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 29, Photosynthetically active radiation (400 - 700 nm) photon flux at a level **   | 0-33-050 | Global GTSPP quality flag |   **   | 1-11-000 | Delayed replication of 11 descriptors |   **   | 0-31-001 | Delayed descriptor replication factor | Number of profile sections (e.g. descending, horizontal, ascending) **   | 0-22-056 | Direction of profile | 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal) **   | 1-08-000 | Delayed replication of 8 descriptors |   **   | 0-31-002 | Extended delayed descriptor replication factor |   **   | 3-01-011 | Year, month, day |   **   | 3-01-013 | Hour, minute, second |   **   | 3-01-021 | Latitude / Longitude (high accuracy) |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 20, position **   | 0-33-050 | Global GTSPP quality flag |   **   | 0-41-005 | Turbidity |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 30, turbidity at a level **   | 0-33-050 | Global GTSPP quality flag |   **   | 1-15-000 | Delayed replication of 15 descriptors |   **   | 0-31-001 | Delayed descriptor replication factor | Number of profile sections (e.g. descending, horizontal, ascending) **   | 0-22-056 | Direction of profile | 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal) **   | 1-12-000 | Delayed replication of 12 descriptors |   **   | 0-31-002 | Extended delayed descriptor replication factor |   **   | 3-01-011 | Year, month, day |   **   | 3-01-013 | Hour, minute, second |   **   | 3-01-021 | Latitude / Longitude (high accuracy) |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 20, position **   | 0-33-050 | Global GTSPP quality flag |   **   | 2-01-135 | Add 7 bits to width |   **   | 2-02-131 | Add 3 to scale |   **   | 0-13-080 | Water pH |   **   | 2-02-000 | Cancel change scale |   **   | 2-01-000 | Cancel change width |   **   | 0-08-080 | Qualifier for GTSPP quality flag | 21, water pH **   | 0-33-050 | Global GTSPP quality flag |  
vturpin commented 1 year ago

Please colleague, consider this as a clarification of what COULD be the reduced Table D from the perspective of someone discovering BUFR format but "expert" in gliders. (** mean out of the reduced table D). This is only to answer @castelao request. Nothing more. This should not bring confusion to the discussion, sorry if that is the case.

castelao commented 1 year ago

I think there is a strategic decision to be made here before any technical solution. I understand and agree with the urge to have this template included in the next round. The question is how valuable is to put effort at this moment to add more variables beyond the CTD plus the vertically averaged current? There is no reason to put time into more variables if it's not believed to be valuable at this moment. @vturpin and @kbailey-noaa might be better suited to decide that.

A few comments that might help (please, don't be shy to correct me if there is any information wrong here):

I'm not sure how much I'm actually helping here, so I'll step back and get out of your way. @david-i-berry and @amilan17 , if you have the time, I created some sample datasets and would be very much interested in learning if my binaries have a consistent structure, or if I done any logical mistake. If you have the time for that, it would be probably better done outside this issue to avoid confusion.

vturpin commented 1 year ago

The priority is to turn this BUFR format operational by May2023.

I think this template, even if only for CTD and current at the moment, give us a nice framework to work with in the future. It will not impact data flow much, but it make an important difference on the long term (I think). It will give us the capacity to evolve independantly from other format.

I agree that CHLA and O2 can be mature for some groups and usefull for some applications, eventhough I understood that from last Argo Data Management Team meeting, the risk related to push imature data on GTS is to be taken seriously as it brings discredit on the program itself (I discovered that aspect during the meeting two weeks ago).

I was not aware of RT transmission of current at depth. Great ! Let's keep it in the format as it is "pretty strait forward".

If we can define a format for T, S, current (average and at depth), CHLA and O2, in time, I'll be more than happy. If not, then we should stick to T, S, Current. Keeping in mind the "Perferct is the enemy of good".

Thanks @castelao for providing the data sets for validation. That's great.

david-i-berry commented 1 year ago

Based on the above comments I've simplified the template to the proposal below, keeping only the CTD elements. In the next period I would then recommend developing the additional sequences to add in the ChlA, O2, PAR etc. It should then be possible to use these additional sequences with the Argo data.

Revised proposal

New Table B Entries

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-42-016 Sea water potential density referenced to sea surface kg m-3 5 90000000 24

New Table D Entries

Category 15: Oceanographic report sequences

Table Reference Table References Element Name Description
3-15-012 Ocean glider trajectory profile sequence Sequence for reporting observations from a single glider profile
3-01-150 (WIGOS identifier)
2-01-129 Change data width Increase width of following elements by 1 bit
0-01-087 WMO marine observing platform extended identifier
2-01-000 Change data width Cancel increase in width
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
0-22-032 Speed of sea surface current
0-22-005 Direction of sea-surface current
0-08-021 Time significance Set to 2, time averaged
0-04-025 Time period or displacement Duration of dive
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
1-26-000 Delayed replication of 26 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-23-000 Delayed replication of 23 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

## Summary of latest changes Simplification following above discussion to only include CTD measurements in first version of the profile. Additional data will be added by defining additional short sequences to be appended, following the pratices developed in the Argo community.
david-i-berry commented 1 year ago

If the above is acceptable hopefully we can get the validation completed by Christmas.

david-i-berry commented 1 year ago

I'm just looking at the EGO NetCDF format (https://repository.oceanbestpractices.org/bitstream/handle/11329/1253/71648.pdf). In this there are 7 different glider phases listed, in BUFR we only have 3 possible. Will this be an issue?

EGO NetCDF image

BUFR image

vturpin commented 1 year ago

What are the phase in BUFR ?

  1. Ascent
  2. Descent
  3. Drift (I guess?)

However, I don't think it is an issue. Only the corresponding phase can be used and the rest ignored.

david-i-berry commented 1 year ago

Hopefully final version:

Revised proposal

New Table B Entries

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-42-016 Sea water potential density referenced to sea surface kg m-3 5 -10000000 24

New Table D Entries

Category 15: Oceanographic report sequences

Table Reference Table References Element Name Description
3-15-012 Ocean glider trajectory profile sequence Sequence for reporting observations from a single glider profile
3-01-150 (WIGOS identifier)
2-01-129 Change data width Increase width of following elements by 1 bit
0-01-087 WMO marine observing platform extended identifier
2-01-000 Change data width Cancel increase in width
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
0-22-032 Speed of sea surface current
0-22-005 Direction of sea-surface current
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
0-08-021 Time significance Set to 2, time averaged
0-04-025 Time period or displacement Duration of dive
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
1-26-000 Delayed replication of 26 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-23-000 Delayed replication of 23 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

## Summary of latest changes - Change of reference for new potential density element, noting that the potential density is density at reference salinity, temperature and pressure - 1000. - Move of date/time in profile mean to before time significance and time period (I am unsure why I moved this originally, I no longer have access to the emails where it would have been discussed). Test data (json) and encoded (BUFR) attached, code to follow. The test data have been extracted from https://www.bodc.ac.uk/data/bodc_database/gliders/ and the AlterEco2 deployment 496 of Melonhead. A straight conversion for salinity from 0/00 to psu was made as salinity in BUFR is in 0/00 whilst that in the test file in psu. **This needs to be checked.** [test-data.zip](https://github.com/wmo-im/BUFR4/files/10374337/test-data.zip)
david-i-berry commented 1 year ago

Happy new year, would any of you (@vturpin @castelao @sebvi @marianmajan-ibl @marijanacrepulja @SibylleK ) be able to help with the validation of the data in the last post above? I've used the eccodes python library for encoding.

david-i-berry commented 1 year ago

Final modification:

Common code table C-13

Add

Data category : 31 Oceanographic data International data sub-category: 8 Glider data

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference-10.01.2023 notes:

@david-i-berry updated the team on the hopefully final proposal; @david-i-berry will add issue to CCT repo on new C-13 value, update issue summary with test data file and update branches. @marijanacrepulja and @SibylleK will validate the sample data;

test-data.zip

SibylleK commented 1 year ago

Here the output of the DWD BUFR software using the BUFR tables of the branch 16-new-table-d-sequence-for-reporting-observations-from-ocean-gliders: glider-test-v40.bufr.DWDout.txt

Apart from the representation of the scaled values, the values are the same as in the melonhead.json file. Whereat the values of "depthBelowWaterSurface" (007062) and "oceanographicWaterPressure" (022065) in the json file look very strange.

By the way, the accuracy of the water pressure is 10 hPa (i.e. 0,1 decibar) and decimeter of the depth below sea/water surface. Is this sufficient?

david-i-berry commented 1 year ago

Sorry, there may be spurious precision in the json file due to dumping from the netcdf to json. Otherwise the pressures and depths agree with each other and are in the expected range.

Numerical precision of 10 hPa or 0.1m should be sufficient.

marijanacrepulja commented 1 year ago

@amilan17 Here is the output of the ECMWF BUFR software using the BUFR tables of the branch 16-new-table-d-sequence-for-reporting-observations-from-ocean-gliders: glider-test.bufr.ECMWFout.txt

I did cross-check values from the DWD output provided by @SibylleK. The decoded values are the same.

I believe we can finalise validation.

david-i-berry commented 1 year ago

@marijanacrepulja @SibylleK thanks!

vturpin commented 1 year ago

Sorry to ask this here (maybe again), but I wonder when is the next "FT approuval procedure". I've been asked by NOAA partners about the release of the updated WMO Manual on Codes. Thanks.

amilan17 commented 1 year ago

@vturpin This proposal and others are currently available for review by National Focal Points until April 28. So far, we've received no comments, so I expect that it will be accepted. The intended implementation date for exchange of data using these new sequences is 31 May 2023. The WMO approval and publication of the machine-readable codes may be available before then. I hope this helps.

vturpin commented 3 months ago

@amilan17 I just check the Vol 1.2 of manual of code and I still don't see the mention of the sequence dedicated to glider 3.15.012. Any idea when this will be published ?

Also, you mention the "WMO approval and publication of the machine redable code" is this available ? And where ?

Finally I struggle finding the place, apart from this issue, where I can see a the version of the sequence that has been accepted. I would like to guide some users to a more "official" documentation as they start to implement the BUFR template but I feel lost at the moment.

Thanks for your help Anna

amilan17 commented 3 months ago

@vturpin They will be included in the next publication of the Manual on Codes, which is being prepared for publication right now but I don't know when it will be finalized and available from the WMO library. Until then you can refer to this provisional copy: https://wmoomm.sharepoint.com/:b:/s/wmocpdb/EW-YM8uJZlRJpe0WPrlyVGEB6Ll_OAwZuVlW_CUJiWpJ7Q?e=Aa1Dnj I hope this helps.