search

wmo-im / BUFR4

BUFR edition 4
MIT License
27 stars 9 forks source link

New Table D sequence for reporting observations from ocean gliders #16

Closed david-i-berry closed 1 year ago

david-i-berry commented 4 years ago

Authors

Summary and purpose

The Ocean Gliders Data Management Task Team (OGDMTT) proposes a new BUFR sequence to report the full suite of glider observations along a trajectory profile.

Action proposed

The team is kindly asked to review and approve the contents for inclusion within the next update to the WMO Manual on Codes.

Discussions

Ocean glider data is currently submitted to the GTS using the Argo BUFR sequence (3-15-003) but this is not ideal. For example, the Argo sequence does not allow the along track positions to be reported as part of the trajectory profile and a number of parameters are missing. This new sequence seeks to address those deficiencies.

Detailed proposal (updated 10/01/2023)

New Table B Entries

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-42-016 Sea water potential density referenced to sea surface kg m-3 5 -10000000 24

New Table D Entries

Category 15: Oceanographic report sequences new sequence for Ocean glider trajectory profile

Table Reference Table References Element Name Description
3-15-012 Sequence for reporting observations from a single glider trajectory profile
3-01-150 (WIGOS identifier)
2-01-129 Change data width Increase width of following elements by 1 bit
0-01-087 WMO marine observing platform extended identifier
2-01-000 Change data width Cancel increase in width
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
0-22-032 Speed of sea surface current
0-22-005 Direction of sea-surface current
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
0-08-021 Time significance Set to 2, time averaged
0-04-025 Time period or displacement Duration of dive
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
1-26-000 Delayed replication of 26 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-23-000 Delayed replication of 23 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

See also https://github.com/wmo-im/CCT/issues/122. ## Test data [test-data.zip](https://github.com/wmo-im/BUFR4/files/10383572/test-data.zip)
jbathegit commented 4 years ago

I spotted a couple of typos - the FXY value for "Direction of sea-surface current" should be 0-22-005 (not 0-02-169), and the FXY value for "Speed of sea-surface current" should be 0-22-032 (not 0-22-031).

david-i-berry commented 4 years ago

Thanks, it's hopefully correct now.

chenxiaoxia2019 commented 4 years ago

@DavidBerryNOC I have created a branch for this issue. Could you please check it? BTW, when it comes to adding table B entries/class 41/42, please provide the information of CREX unit/scale/width. Many thanks.

david-i-berry commented 4 years ago

Revision of proposal as part of validation

Table B entries

Table D entries

  1. Removal of redundant descriptors from D sequence (originating centre and sub-centre)
  2. Move of time significance and duration
  3. Correction of number of replicated descriptors
  4. Correction of number of replicated descriptors
  5. Update of element names for new descriptors

Revised proposal

New Table B Entries

Class 41: BUFR / CREX Oceanographic / biogeochemical parameters

Descriptor Name Units Scale Reference Width Notes
0-41-006 Mass concentration of chlorophyll A in sea water kg m-3 9 0 21
0-41-007 Concentration of coloured dissolved organic matter (CDOM) in sea water expressed as equivalent mass fraction of quinine sulfate dihydrate numeric 12 0 21 parts per billion
0-41-008 Volume backwards scattering coefficient of radiative flux in sea water (BBP) m-1 6 0 25
0-41-009 Photosynthetically active radiation (400-700 nm) photon flux mol m-2 s-1 10 0 26

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes
0-42-016 Sea water potential density referenced to sea surface kg m-3 3 1000000 16

New Table D Entries

Category 15: Oceanographic report sequences

Change Table Reference Table References Element Name Description
3-15-012 Ocean glider trajectory profile sequence
3-01-150 (WIGOS identifier)
0-01-087 WMO marine observing platform extended identifier
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
1 ~0-01-033~ ~Identification of originating / generating centre~
1 ~0-01-034~ ~Identification of originating / generating sub-centre~
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
0-22-005 Direction of sea-surface current
0-22-032 Speed of sea surface current
2 0-08-021 Time significance Set to 2, time averaged
2 0-04-025 Time period or displacement Duration of dive
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
2 ~0-08-021~ ~Time significance~ ~Set to 2, time averaged~
2 ~0-04-025~ ~Time period or displacement~ ~Duration of dive~
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
3 1-54-000 Delayed replication of 54 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
4 1-51-000 Delayed replication of 51 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag
0-22-031 Speed of current
0-08-080 Qualifier for GTSPP quality flag 14, sea / water current speed at a level
0-33-050 Global GTSPP quality flag
0-22-004 Direction of current
0-08-080 Qualifier for GTSPP quality flag 15, sea / water current direction at a level
0-33-050 Global GTSPP quality flag
0-22-188 Dissolved oxygen
0-08-080 Qualifier for GTSPP quality flag 16, dissolved oxygen at a level
0-33-050 Global GTSPP quality flag
0-41-006 Mass concentration of chlorophyll A in sea water
0-08-080 Qualifier for GTSPP quality flag 27, mass concentration of chlorophyll A at a level
0-33-050 Global GTSPP quality flag
5 0-41-007 Concentration of coloured dissolved organic matter (CDOM) in sea water expressed as equivalent mass fraction of quinine sulfate dihydrate
0-08-080 Qualifier for GTSPP quality flag 28 concentration of coloured dissolved organic matter in sea water at a level
0-33-050 Global GTSPP quality flag
1-04-000 Delayed replication of 4 descriptors
0-31-001 Delayed descriptor replication factor
0-02-071 Spectrographic wavelength
5 0-41-008 Volume backwards scattering coefficient of radiative flux in sea water (BBP)
0-08-080 Qualifier for GTSPP quality flag 29, particle beam attenuation coefficient at a level
0-33-050 Global GTSPP quality flag
0-02-071 Spectrographic wavelength
0-41-009 Photosynthetically active radiation (400-700 nm) photon flux
0-08-080 Qualifier for GTSPP quality flag 30, Photosynthetically active radiation (400 - 700 nm) photon flux at a level
0-33-050 Global GTSPP quality flag
0-41-005 Turbidity
0-08-080 Qualifier for GTSPP quality flag 31, turbidity at a level
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
* 27Mass concentration of chlorophyll A at a level
* 28Concentration of coloured dissolved organic matter in sea water at a level
* 29Particle beam attenuation coefficient at a level
* 30Photosynthetically active radiation (400 - 700 nm) photon flux at a level
* 31Turbidity at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

david-i-berry commented 4 years ago

@DavidBerryNOC I have created a branch for this issue. Could you please check it? BTW, when it comes to adding table B entries/class 41/42, please provide the information of CREX unit/scale/width. Many thanks.

@chenxiaoxia2019 revised versions of tables attached - substantial changes made to Table D due to errors in branch.

BUFRCREX_TableB_en.txt BUFR_TableD_en.txt

david-i-berry commented 4 years ago

@chenxiaoxia2019 I've added CREX width scale below - please let me know it you need further info.

Revision of proposal as part of validation

Further modifications to include CREX scale / width plus increase of width for dissolved oxygen using operators.

Test files and revised tables attached.

Table B entries

Table D entries

  1. Removal of redundant descriptors from D sequence (originating centre and sub-centre)
  2. Swap order of surface current speed and direction to consistent with the other occurrences
  3. Move of time significance and duration
  4. Correction of number of replicated descriptors
  5. Correction of number of replicated descriptors
  6. Increase width by 2 for dissolved oxygen to accommodate high near surface values (new max value 1048 micro mol / kg)
  7. Update of element names for new descriptors

Revised proposal

New Table B Entries

Class 41: BUFR / CREX Oceanographic / biogeochemical parameters

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-41-006 Mass concentration of chlorophyll A in sea water kg m-3 9 0 21 9 7
0-41-007 Concentration of coloured dissolved organic matter (CDOM) in sea water expressed as equivalent mass fraction of quinine sulfate dihydrate numeric 12 0 21 parts per billion 12 7
0-41-008 Volume backwards scattering coefficient of radiative flux in sea water (BBP) m-1 6 0 25 6 8
0-41-009 Photosynthetically active radiation (400-700 nm) photon flux mol m-2 s-1 10 0 26 10 8

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-42-016 Sea water potential density referenced to sea surface kg m-3 5 1000000 23 5 7

New Table D Entries

Category 15: Oceanographic report sequences

Change Table Reference Table References Element Name Description
3-15-012 Ocean glider trajectory profile sequence
3-01-150 (WIGOS identifier)
0-01-087 WMO marine observing platform extended identifier
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
1 ~0-01-033~ ~Identification of originating / generating centre~
1 ~0-01-034~ ~Identification of originating / generating sub-centre~
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
2 0-22-032 Speed of sea surface current
0-22-005 Direction of sea-surface current
2 ~0-22-032~ ~Speed of sea surface current~
3 0-08-021 Time significance Set to 2, time averaged
3 0-04-025 Time period or displacement Duration of dive
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3 ~0-08-021~ ~Time significance~ ~Set to 2, time averaged~
3 ~0-04-025~ ~Time period or displacement~ ~Duration of dive~
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
4 1-56-000 Delayed replication of 56 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
5 1-53-000 Delayed replication of 53 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag
0-22-031 Speed of current
0-08-080 Qualifier for GTSPP quality flag 14, sea / water current speed at a level
0-33-050 Global GTSPP quality flag
0-22-004 Direction of current
0-08-080 Qualifier for GTSPP quality flag 15, sea / water current direction at a level
0-33-050 Global GTSPP quality flag
6 2-01-130 Change data width Add 2 bits to data width
0-22-188 Dissolved oxygen
6 2-01-000 Change data width cancel operation
0-08-080 Qualifier for GTSPP quality flag 16, dissolved oxygen at a level
0-33-050 Global GTSPP quality flag
0-41-006 Mass concentration of chlorophyll A in sea water
0-08-080 Qualifier for GTSPP quality flag 27, mass concentration of chlorophyll A at a level
0-33-050 Global GTSPP quality flag
7 0-41-007 Concentration of coloured dissolved organic matter (CDOM) in sea water expressed as equivalent mass fraction of quinine sulfate dihydrate
0-08-080 Qualifier for GTSPP quality flag 28 concentration of coloured dissolved organic matter in sea water at a level
0-33-050 Global GTSPP quality flag
1-04-000 Delayed replication of 4 descriptors
0-31-001 Delayed descriptor replication factor
0-02-071 Spectrographic wavelength
7 0-41-008 Volume backwards scattering coefficient of radiative flux in sea water (BBP)
0-08-080 Qualifier for GTSPP quality flag 29, particle beam attenuation coefficient at a level
0-33-050 Global GTSPP quality flag
0-02-071 Spectrographic wavelength
0-41-009 Photosynthetically active radiation (400-700 nm) photon flux
0-08-080 Qualifier for GTSPP quality flag 30, Photosynthetically active radiation (400 - 700 nm) photon flux at a level
0-33-050 Global GTSPP quality flag
0-41-005 Turbidity
0-08-080 Qualifier for GTSPP quality flag 31, turbidity at a level
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
* 27Mass concentration of chlorophyll A at a level
* 28Concentration of coloured dissolved organic matter in sea water at a level
* 29Particle beam attenuation coefficient at a level
* 30Photosynthetically active radiation (400 - 700 nm) photon flux at a level
* 31Turbidity at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

samples.zip BUFR_TableD_en.txt BUFRCREX_TableB_en.txt

david-i-berry commented 4 years ago

I've just added the help wanted - if anyone is able to decode the samples to help the validation that would be appreciated. Please note that the tables in the branch are incorrect and that tables attached to the the previous comment (https://github.com/wmo-im/BUFR4/issues/16#issuecomment-646545399) need to be used.

david-i-berry commented 4 years ago

@DavidBerryNOC I have created a branch for this issue. Could you please check it? BTW, when it comes to adding table B entries/class 41/42, please provide the information of CREX unit/scale/width. Many thanks.

@chenxiaoxia2019 Please see comments above in reference to the CREX unit/scale/width, units are the same as for BUFR.

I've also updated my versions of the tables from the branch (attached above) - what would be the best way to update these in the branch? Should I do a pull request or will that be handled by the secretariat?

chenxiaoxia2019 commented 4 years ago

@DavidBerryNOC Hi, David, I will update all the revisions in the branch. I will get you updated when I finish it.

david-i-berry commented 4 years ago

The new updates to the branch remain to be checked, plus there is a further change required. Noting significant number of changes to proposal, propose delaying to FT-2021-01 to allow time for branch to be checked and revised width and scale of new B descriptors to be checked by ocean community.

amilan17 commented 4 years ago

Please validate by December @marijanacrepulja @marianmajan-ibl

marianmajan-ibl commented 4 years ago

Hello @amilan17 and @DavidBerryNOC ,

I was trying to validate the sequence 3 15 012 from the branch issune-16. However the content does not seem to be right (its descriptors seem to be just incremented 001087):

15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,301150,(WIGOS identifier),,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001087,WMO marine observing platform extended identifier,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001088,Long station or site name,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001089,Agency in charge of operating the observing platform,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001090,Data collection and/or location system,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001091,Identification of originating / generating centre,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001092,Identification of originating / generating sub-centre,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001093,Observing platform manufacturer's model,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001094,Observing platform manufacturer's serial number,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001095,Time significance,"Set to 25, nominal reporting time",,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001096,"(Year, month, day)",,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001097,"(Hour, minute, second)",,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001098,(Latitude / longitude (high accuracy)),,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001099,"True heading of aircraft, ship or other mobile platform",,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001100,Anemometer type,"Set to 2, wind observation through ambient noise (WOTAN)",,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001101,Wind speed,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001102,Wind direction,,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001103,Anemometer type,Set to missing / cancel previous value,,Operational
15,Oceanographic report sequences,315012,(Ocean glider trajectory profile sequence),,001104,Direction of sea-surface current,,,Operational

Am I missing something?

Best regards, Marian

marijanacrepulja commented 4 years ago

I was able to decode a sample BUFR message using the tables provided in Dave's latest comment. The entries are not in the branch yet. We are also missing the code_flag table update with additional values.

david-i-berry commented 3 years ago

I'm afraid we need to delay further, there is an outstanding item to confirm the scale, width and offset required for the new table B entries.

castelao commented 3 years ago

Hi, I'm sorry to jump into this discussion. I would like to suggest including pH in this template. This is a relatively new sensor for autonomous platforms but highly desirable for realtime transmission, hence would benefit from BUFR. This suggestion would require:

Changes: ... 1-56-000 -> 1-61-000 ... 1-53-000 -> 1-58-000 ...

Addition in the end:

1-03-000 0-31-000 0-33-050 Global GTSPP quality flag; width 2 0-13-080 Water pH 0-08-080 Qualifier for GTSPP quality flag; width 6 (Would require a new entry on 0-08-080? Maybe code 32?

Note that I used 0-31-000 adding one extra bit to potentially save 18 bits (2 + 10 + 6). On that point, I think that all variables after the 0-42-016 are not regular measurements for underwater gliders, and might benefit from space-saving with 0-31-000 switcher. Especially for 0-02-071 & 0-41-008.

amilan17 commented 3 years ago

@DavidBerryNOC 

It appears that the previous issue branch "issune-16" was deleted. 

Please update this new issue branch:  https://github.com/wmo-im/BUFR4/tree/issue-16

david-i-berry commented 3 years ago

There is still an outstanding question on the width and scale factor for two of the new parameters. Once we have resolved those I will update the issue branch and look to redo the validation.

david-i-berry commented 3 years ago

Comments received via email on scale, width and offset of new parameters.

Descriptor Name Units Scale Reference Width Notes Min Max Precision Giorgio Dall'Olmo Argo IOOS/Quartod World Ocean database
0-41-006 Mass concentration of chlorophyll A in sea water kg m-3 9 0 21   0 0.00209715 1.0000E-09 For open ocean and coastal waters a concentration of 100 mg/m3 is extremely high.Here the max is set at 2097 mg/m3. It sounds a little high to me, but I guess it's a very safe upper bound. https://archimer.ifremer.fr/doc/00243/35385/CHLA in range [?0.1, 50] mg/m3(#RD4) https://ioos.noaa.gov/ioos-in-action/phytoplankton/"Gross range/threshold tests to be specified by operator" https://www.ncei.noaa.gov/data/oceans/woa/WOD/DOC/wodreadme.pdfp89 for CHLA
0-41-007 Concentration of coloured dissolved organic matter (CDOM) in sea water expressed as equivalent mass fraction of quinine sulfate dihydrate numeric 12 0 21 parts per billion 0 2.0972E-06 1.0000E-12 Not sure about this, I guess you could use calibration certificates for CDOM sensors to define this. (I do not have any handy) https://archimer.ifremer.fr/doc/00298/40879/Struggling to find limits defined in Argo https://ioos.noaa.gov/ioos-in-action/oceanic-optics/"Gross range/threshold tests to be specified by operator"  
0-41-008 Volume backwards scattering coefficient of radiative flux in sea water (BBP) m-1 6 0 25   0 33.55443 1.0000E-06 For BBP a value of 0.1 1/m is extremely high, so again a max of 33 1/m seems a very high upper bound. https://archimer.ifremer.fr/doc/00298/40879/BBP700 in range [?0.000025, 0.1] m-1(#RD5)BBP532 in range [?0.000005, 0.1] m-1(#RD5) https://ioos.noaa.gov/ioos-in-action/oceanic-optics/"Gross range/threshold tests to be specified by operator"  
0-41-009 Photosynthetically active radiation (400-700 nm) photon flux mol m-2 s-1 10 0 26   0 0.00671089 1.0000E-10 From what I know a max PAR of 0.001 mol/m2/s would be enough. So again the max you set is a very high upper bound. https://archimer.ifremer.fr/doc/00298/40879/Struggling to find limits defined in Argo https://ioos.noaa.gov/ioos-in-action/oceanic-optics/"Gross range/threshold tests to be specified by operator"  
0-42-016 Sea water potential density referenced to sea surface kg m-3 3 1000000 16   1000 1065.534 1.0000E-03 I'm not an expert on this. https://archimer.ifremer.fr/doc/00228/33951/Does not cover potential density https://ioos.noaa.gov/ioos-in-action/temperature-salinity/"Gross range/threshold tests to be specified by operator" https://www.ncei.noaa.gov/data/oceans/woa/WOD/DOC/wodreadme.pdf
amilan17 commented 3 years ago

@DavidBerryNOC - branch is rebased and ready for further editing as needed.

david-i-berry commented 3 years ago

Revision of proposal as part of validation

Table B entries

Table D entries

  1. Separation of elements into more optional sections using delayed replication. this gives more flexibility over what is reported, many gliders only report conductivity, temperature, depth.

Other comments

One further change may be required depending on whether depth / pressure is required for each optional profile or whether this will be interpolated from the CTD section.

Revised proposal

New Table B Entries

Class 41: BUFR / CREX Oceanographic / biogeochemical parameters

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-41-006 Mass concentration of chlorophyll A in sea water kg m-3 9 0 17 9 7
0-41-007 Volume backwards scattering coefficient of radiative flux in sea water (BBP) m-1 6 0 17 6 8
0-41-008 Photosynthetically active radiation (400-700 nm) photon flux mol m-2 s-1 10 0 24 10 8

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-42-016 Sea water potential density referenced to sea surface kg m-3 5 100000000 23 5 7

New Table D Entries

Category 15: Oceanographic report sequences

Table Reference Table References Element Name Description
3-15-012 Ocean glider trajectory profile sequence
3-01-150 (WIGOS identifier)
0-01-087 WMO marine observing platform extended identifier
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
0-22-032 Speed of sea surface current
0-22-005 Direction of sea-surface current
0-08-021 Time significance Set to 2, time averaged
0-04-025 Time period or displacement Duration of dive
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
1-26-000 Delayed replication of 26 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-23-000 Delayed replication of 23 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag
1-14-000 Delayed replication of 14 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-11-000 Delayed replication of 11 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-22-031 Speed of current
0-08-080 Qualifier for GTSPP quality flag 14, sea / water current speed at a level
0-33-050 Global GTSPP quality flag
0-22-004 Direction of current
0-08-080 Qualifier for GTSPP quality flag 15, sea / water current direction at a level
0-33-050 Global GTSPP quality flag
1-13-000 Delayed replication of 13 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-10-000 Delayed replication of 10 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
2-01-130 Change data width Add 2 bits to data width
0-22-188 Dissolved oxygen
2-01-000 Change data width cancel operation
0-08-080 Qualifier for GTSPP quality flag 16, dissolved oxygen at a level
0-33-050 Global GTSPP quality flag
1-18-000 Delayed replication of 18 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-15-000 Delayed replication of 15 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-006 Mass concentration of chlorophyll A in sea water
0-08-080 Qualifier for GTSPP quality flag 27, mass concentration of chlorophyll A at a level
0-33-050 Global GTSPP quality flag
1-04-000 Delayed replication of 4 descriptors
0-31-001 Delayed descriptor replication factor
0-02-071 Spectrographic wavelength
0-41-007 Volume backwards scattering coefficient of radiative flux in sea water (BBP)
0-08-080 Qualifier for GTSPP quality flag 28, particle beam attenuation coefficient at a level
0-33-050 Global GTSPP quality flag
0-02-071 Spectrographic wavelength cancel
1-14-000 Delayed replication of 18 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-15-000 Delayed replication of 15 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-008 Photosynthetically active radiation (400-700 nm) photon flux
0-08-080 Qualifier for GTSPP quality flag 29, Photosynthetically active radiation (400 - 700 nm) photon flux at a level
0-33-050 Global GTSPP quality flag
1-18-000 Delayed replication of 18 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-15-000 Delayed replication of 15 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-005 Turbidity
0-08-080 Qualifier for GTSPP quality flag 30, turbidity at a level
0-33-050 Global GTSPP quality flag
1-15-000 Delayed replication of 15 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-15-000 Delayed replication of 12 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
2-01-135 Add 7 bits to width
2-02-131 Add 3 to scale
0-13-080 Water pH
2-02-000 Cancel change scale
2-01-000 Cancel change width
0-08-080 Qualifier for GTSPP quality flag 21, water pH
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
* 27Mass concentration of chlorophyll A at a level
* 28Particle beam attenuation coefficient at a level
* 29Photosynthetically active radiation (400 - 700 nm) photon flux at a level
* 30Turbidity at a level
* 31Water pH at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

castelao commented 3 years ago

@DavidBerryNOC , moving the discussions from email to here. In summary:

  1. I believe that there are some typos on the X value of the last replicators:

    • 1-14-000 -> 1-11-000 (PAR)
    • 1-15-000 -> 1-08-000 (PAR)
    • 1-18-000 -> 1-11-000 (Turbidity)
    • 1-15-000 -> 1-08-000 (Turbidity)
    • 1-15-000 (pH, is correct)
    • 1-15-000 -> 1-12-000 (pH)

  2. I believe that we need two bytes for the replicator of number of profiles.

  3. I think it should be OK to have one single top level number of profiles replicator + profile direction. But keep the individual replicators per sensor

Thank you!

castelao commented 3 years ago

About item 3 in my previous post, I don't have yet a clear opinion on what to suggest. The advantage of a single upper-level replicator for profiles is to reinforce consistency between the sensors, which when available would probably be functional on every dive. But that would require all sensors that are not present to have a zero per dive in the delayed replicator, i.e., as many zeros as # of missing sensors times # of profiles. On the other side, the original structure proposed by @DavidBerryNOC would permit one single zero in the top-level replicator per missing sensor, informing that there are no profiles with that variable.

I changed my opinion since my previous post and I'm now more inclined to the original @DavidBerryNOC 's structure. I.e., ignore item 3, but I think that the items 1 & 2 are still an issue.

david-i-berry commented 3 years ago

I've made the changes for (1) and have included the update below.

For (2), the original plan was for each BUFR message to contain the data for a single profile and so we used the delayed descriptor replication factor so we could report the different phases of a single profile. At the moment we can have up to 255 different segments for a single profile but we can increase that if needed. I've update the description to state that it is for a single profile.

Revised proposal

New Table B Entries

Class 41: BUFR / CREX Oceanographic / biogeochemical parameters

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-41-006 Mass concentration of chlorophyll A in sea water kg m-3 9 0 17 9 7
0-41-007 Volume backwards scattering coefficient of radiative flux in sea water (BBP) m-1 6 0 17 6 8
0-41-008 Photosynthetically active radiation (400-700 nm) photon flux mol m-2 s-1 10 0 24 10 8

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-42-016 Sea water potential density referenced to sea surface kg m-3 5 100000000 23 5 7

New Table D Entries

Category 15: Oceanographic report sequences

Table Reference Table References Element Name Description
3-15-012 Ocean glider trajectory profile sequence Sequence for reporting observations from a single glider profile
3-01-150 (WIGOS identifier)
0-01-087 WMO marine observing platform extended identifier
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
0-22-032 Speed of sea surface current
0-22-005 Direction of sea-surface current
0-08-021 Time significance Set to 2, time averaged
0-04-025 Time period or displacement Duration of dive
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
1-26-000 Delayed replication of 26 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-23-000 Delayed replication of 23 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag
1-14-000 Delayed replication of 14 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-11-000 Delayed replication of 11 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-22-031 Speed of current
0-08-080 Qualifier for GTSPP quality flag 14, sea / water current speed at a level
0-33-050 Global GTSPP quality flag
0-22-004 Direction of current
0-08-080 Qualifier for GTSPP quality flag 15, sea / water current direction at a level
0-33-050 Global GTSPP quality flag
1-13-000 Delayed replication of 13 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-10-000 Delayed replication of 10 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
2-01-130 Change data width Add 2 bits to data width
0-22-188 Dissolved oxygen
2-01-000 Change data width cancel operation
0-08-080 Qualifier for GTSPP quality flag 16, dissolved oxygen at a level
0-33-050 Global GTSPP quality flag
1-18-000 Delayed replication of 18 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
1-15-000 Delayed replication of 15 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-006 Mass concentration of chlorophyll A in sea water
0-08-080 Qualifier for GTSPP quality flag 27, mass concentration of chlorophyll A at a level
0-33-050 Global GTSPP quality flag
1-04-000 Delayed replication of 4 descriptors
0-31-001 Delayed descriptor replication factor
0-02-071 Spectrographic wavelength
0-41-007 Volume backwards scattering coefficient of radiative flux in sea water (BBP)
0-08-080 Qualifier for GTSPP quality flag 28, particle beam attenuation coefficient at a level
0-33-050 Global GTSPP quality flag
0-02-071 Spectrographic wavelength cancel
* 1-11-000 Delayed replication of 11 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
* 1-08-000 Delayed replication of 8 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-008 Photosynthetically active radiation (400-700 nm) photon flux
0-08-080 Qualifier for GTSPP quality flag 29, Photosynthetically active radiation (400 - 700 nm) photon flux at a level
0-33-050 Global GTSPP quality flag
* 1-11-000 Delayed replication of 11 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
* 1-08-000 Delayed replication of 8 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-005 Turbidity
0-08-080 Qualifier for GTSPP quality flag 30, turbidity at a level
0-33-050 Global GTSPP quality flag
1-15-000 Delayed replication of 15 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 1 (descending / downwards profile), 2 (horizontal) or 3 (ascending / upwards profile)
* 1-12-000 Delayed replication of 12 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
2-01-135 Add 7 bits to width
2-02-131 Add 3 to scale
0-13-080 Water pH
2-02-000 Cancel change scale
2-01-000 Cancel change width
0-08-080 Qualifier for GTSPP quality flag 21, water pH
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
* 27Mass concentration of chlorophyll A at a level
* 28Particle beam attenuation coefficient at a level
* 29Photosynthetically active radiation (400 - 700 nm) photon flux at a level
* 30Turbidity at a level
* 31Water pH at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

castelao commented 3 years ago

@DavidBerryNOC , it looks great. Thank you!

For (2), the original plan was for each BUFR message to contain the data for a single profile and so we used the delayed descriptor replication factor so we could report the different phases of a single profile. At the moment we can have up to 255 different segments for a single profile but we can increase that if needed. I've update the description to state that it is for a single profile.

Oh, I'm sorry for the confusion. Having 255 possible phases for a single dive seems plenty for me. Maybe my confusion was on the "Ocean glider trajectory profile sequence", since for CF trajectory profile I would expect a group of profiles from the same trajectory, but of course a single profile is a valid structure. I think that the new description clarifies that. I think it is a good approach as you're doing, one BUFR message per profile (or some people would call it a dive with the multiple phases of that dive).

david-i-berry commented 3 years ago

On test encoding some data for a glider from the Indian Ocean / Bay of Bengal (SG579 from https://www.bodc.ac.uk/data/published_data_library/catalogue/10.5285/996bf53d-5448-297a-e053-6c86abc0b996/) it appears the ranges / upper limits are still wrong for some of the new parameters. The histogram below shows the PAR in the top 10m and the red vertical line shows the upper limit based on a width and scale of 24 and 10 respectively. We also need to change the reference value for the potential density as we can get values down to ~ 990 kg m-3 for warm / fresh water.

I haven't been able to test the values for 0-41-007 (BBP / backscatter) due to a lack of suitable test data, I've been using the UK Glider Data Assembly Centre (https://www.bodc.ac.uk/data/bodc_database/gliders/) but this has back scatter in different units (m-1 sr-1).

I've also been thinking about the profile data a bit more whilst test encoding, we have date, time and location as coordinate variables with each profile section but no z (or depth) coordinate. Do we need to include this in some form?

Screenshot 2021-06-18 at 09 25 05

jbathegit commented 2 years ago

For the same reasons noted in #106, the Table B descriptor 0-01-087 should be encapsulated with a 2-01-129 operator to add one extra bit to its width within this proposed new 3-15-012 Table D sequence. This will allow the encoding of any 7-digit number within 0-01-087; in particular numbers beginning with "89" which are designated for use with gliders by OceanOPS, according to the Guide to WIGOS.

david-i-berry commented 2 years ago

Outstanding items

vturpin commented 2 years ago

Hi, Can we be updated on the status toward validation of this new sequence for Ocean Gliders ? The situation regarding the new glider template on GTS is often raised by the Pis and data managers and members of the OceanGliders program. Thanks for your help.

david-i-berry commented 2 years ago

We're still waiting for the outstanding items above, mainly the test data. Once we have that we can proceed with the valuation but the provision of the test data is with the glider community.

kbailey-noaa commented 2 years ago

Thanks @vturpin for asking this - I was wondering the same thing (I'm with US IOOS).

Is the OceanGlider program planning to submit a test dataset? For what it's worth we have the Glider DAC ERDDAP that supplies the U.S. glider profiles that are packaged into the drifting buoy BUFR template, if that helps.
Just hoping to push this along and see how I can help!

vturpin commented 2 years ago

Don't know realy who will provide the test data. I also assume it should be raw data from a glider with PAR sensor and BACKSCATTER sensor.

I know that IMOS is using a sensor called "OCR 504 ICSW" that is a radiometer (measuring irradiance at different wave length), and BACKSCATERMETER see here : https://s3-ap-southeast-2.amazonaws.com/imos-data/IMOS/ANFOG/slocum_glider/BassStrait20220422/IMOS_ANFOG_BCEOPSTUV_20220422T013221Z_SL286_FV01_timeseries_END-20220425T174903Z.nc

And OOI too, see here : https://gliders.ioos.us/erddap/info/cp_380-20211215T0736/index.html

So if those data set correspond to what is expected to run the test, we could ask a subset of raw data from those missions and move on. If someone can verify in the data quickly I can ask the people to send raw data. But I am not expert enough to say that those data sets are appropriate to run the test.

I also do not want to by pass anyone here.

justinbuck commented 2 years ago

@vturpin @david-i-berry Glider deployment 556 (CAMPUS) https://www.bodc.ac.uk/data/bodc_database/gliders/ may meet the needs of testing. The data are downloadable in raw and netcdf formats from the web page.

vturpin commented 2 years ago

I had the following feedback from John Turton that I am sharing with you for modification of the template :
"I am just running through the draft template on GitHub for checking/updating my glider BUFR decoding code and noticed one (minor) error.

For 0-22-056 it should be 0 for ascending (3 is “11” so denotes missing data).

And I think the 1-04-000 replication for the BBP section (loop) should be 1-05-000 as 5 descriptors follow the delayed description replication factor"

justinbuck commented 2 years ago

@david-i-berry I am not sure who we need to notify with the feedback from Jon in the post above, could you advise please?

david-i-berry commented 2 years ago

For 0-22-056 it should be 0 for ascending (3 is “11” so denotes missing data).

This will be corrected but doesn't impact on the Table D sequence itself, more its use and potential incorrect encoding.

And I think the 1-04-000 replication for the BBP section (loop) should be 1-05-000 as 5 descriptors follow the delayed description replication factor

The 1-04-000 is correct, the 5th descriptor (0-02-071, Spectrographic wavelength) cancels that set within the loop but does not need to be in the loop itself.

david-i-berry commented 2 years ago

Revised proposal

New Table B Entries

Class 41: BUFR / CREX Oceanographic / biogeochemical parameters

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-41-006 Mass concentration of chlorophyll A in sea water kg m-3 9 0 17
0-41-007 Volume backwards scattering coefficient of radiative flux in sea water (BBP) m-1 6 0 17
0-41-008 Photosynthetically active radiation (400-700 nm) photon flux mol m-2 s-1 10 0 26

Class 42: BUFR / CREX Oceanographic Elements

Descriptor Name Units Scale Reference Width Notes CREX scale CREX width
0-42-016 Sea water potential density referenced to sea surface kg m-3 5 90000000 24

New Table D Entries

Category 15: Oceanographic report sequences

Table Reference Table References Element Name Description
3-15-012 Ocean glider trajectory profile sequence Sequence for reporting observations from a single glider profile
3-01-150 (WIGOS identifier)
2-01-129 Change data width Increase width of following elements by 1 bit
0-01-087 WMO marine observing platform extended identifier
2-01-000 Change data width Cancel increase in width
0-01-019 Long station or site name
0-01-036 Agency in charge of operating the observing platform
0-02-148 Data collection and/or location system
0-01-085 Observing platform manufacturer's model
0-01-086 Observing platform manufacturer's serial number
0-08-021 Time significance Set to 25, nominal reporting time
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-11-104 True heading of aircraft, ship or other mobile platform
0-02-169 Anemometer type Set to 2, wind observation through ambient noise (WOTAN)
0-11-002 Wind speed
0-11-001 Wind direction
0-02-169 Anemometer type Set to missing / cancel previous value
0-22-032 Speed of sea surface current
0-22-005 Direction of sea-surface current
0-08-021 Time significance Set to 2, time averaged
0-04-025 Time period or displacement Duration of dive
3-01-011 (Year, month, day)
3-01-013 (Hour, minute, second)
3-01-021 (Latitude / longitude (high accuracy))
0-22-031 Speed of current
0-22-004 Direction of current
0-08-021 Time significance Set to missing / cancel previous value
0-05-068 Profile number
0-01-079 Unique identifier for profile
1-26-000 Delayed replication of 26 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-23-000 Delayed replication of 23 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-07-062 Depth below water surface
0-08-080 Qualifier for GTSPP quality flag 13, water depth at a level
0-33-050 Global GTSPP quality flag
0-22-065 Water pressure
0-08-080 Qualifier for GTSPP quality flag 10, water pressure at a level
0-33-050 Global GTSPP quality flag
0-22-045 Sea/water temperature
0-08-080 Qualifier for GTSPP quality flag 11, water temperature at a level
0-33-050 Global GTSPP quality flag
0-22-066 Water conductivity
0-08-080 Qualifier for GTSPP quality flag 25, water conductivity at a level
0-33-050 Global GTSPP quality flag
0-22-064 Salinity
0-08-080 Qualifier for GTSPP quality flag 12, salinity at a level
0-33-050 Global GTSPP quality flag
0-42-016 Sea water potential density referenced to sea surface
0-08-080 Qualifier for GTSPP quality flag 26, sea water potential density at a level
0-33-050 Global GTSPP quality flag
1-14-000 Delayed replication of 14 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-11-000 Delayed replication of 11 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-22-031 Speed of current
0-08-080 Qualifier for GTSPP quality flag 14, sea / water current speed at a level
0-33-050 Global GTSPP quality flag
0-22-004 Direction of current
0-08-080 Qualifier for GTSPP quality flag 15, sea / water current direction at a level
0-33-050 Global GTSPP quality flag
1-13-000 Delayed replication of 13 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-10-000 Delayed replication of 10 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
2-01-130 Change data width Add 2 bits to data width
0-22-188 Dissolved oxygen
2-01-000 Change data width cancel operation
0-08-080 Qualifier for GTSPP quality flag 16, dissolved oxygen at a level
0-33-050 Global GTSPP quality flag
1-18-000 Delayed replication of 18 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-15-000 Delayed replication of 15 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-006 Mass concentration of chlorophyll A in sea water
0-08-080 Qualifier for GTSPP quality flag 27, mass concentration of chlorophyll A at a level
0-33-050 Global GTSPP quality flag
1-04-000 Delayed replication of 4 descriptors
0-31-001 Delayed descriptor replication factor
0-02-071 Spectrographic wavelength
0-41-007 Volume backwards scattering coefficient of radiative flux in sea water (BBP)
0-08-080 Qualifier for GTSPP quality flag 28, particle beam attenuation coefficient at a level
0-33-050 Global GTSPP quality flag
0-02-071 Spectrographic wavelength cancel
1-11-000 Delayed replication of 11 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-08-000 Delayed replication of 8 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-008 Photosynthetically active radiation (400-700 nm) photon flux
0-08-080 Qualifier for GTSPP quality flag 29, Photosynthetically active radiation (400 - 700 nm) photon flux at a level
0-33-050 Global GTSPP quality flag
1-11-000 Delayed replication of 11 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-08-000 Delayed replication of 8 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
0-41-005 Turbidity
0-08-080 Qualifier for GTSPP quality flag 30, turbidity at a level
0-33-050 Global GTSPP quality flag
1-15-000 Delayed replication of 15 descriptors
0-31-001 Delayed descriptor replication factor Number of profile sections (e.g. descending, horizontal, ascending)
0-22-056 Direction of profile 0 (ascending / upwards profile), 1 (descending / downwards profile), or 2 (horizontal)
1-12-000 Delayed replication of 12 descriptors
0-31-002 Extended delayed descriptor replication factor
3-01-011 Year, month, day
3-01-013 Hour, minute, second
3-01-021 Latitude / Longitude (high accuracy)
0-08-080 Qualifier for GTSPP quality flag 20, position
0-33-050 Global GTSPP quality flag
2-01-135 Add 7 bits to width
2-02-131 Add 3 to scale
0-13-080 Water pH
2-02-000 Cancel change scale
2-01-000 Cancel change width
0-08-080 Qualifier for GTSPP quality flag 21, water pH
0-33-050 Global GTSPP quality flag

Additions to code tables

0-08-080
Qualifier for GTSPP quality flag

Code figureMeaning
0Total water pressure profile
1Total water temperature profile
2Total water salinity profile
3Total water conductivity profile
4Total water depth
5–9Reserved
10Water pressure at a level
11Water temperature at a level
12Salinity at a level
13Water depth at a level
14Sea/water current speed at a level
15Sea/water current direction at a level
16Dissolved oxygen at a level
17–19Reserved
20Position
* 25Water conductivity at a level
* 26Sea water potential density referenced to sea surface at a level
* 27Mass concentration of chlorophyll A at a level
* 28Particle beam attenuation coefficient at a level
* 29Photosynthetically active radiation (400 - 700 nm) photon flux at a level
* 30Turbidity at a level
* 31Water pH at a level
32-62Reserved
63Missing value

New entries (25 - 31) indicated by *.

## Summary of latest changes ### Table B entries - 0-41-008, width changed from 24 to 26 to accommodate higher values seen near the surface for PAR - 0-42-016, reference and width changed from 100000000 to 90000000 and 23 to 24 respectively to allow densities below 1000 kg m-3 ### Table D entry - Addition of 2-01-129 wrapping 'WMO marine observing platform extended identifier' to allow greater maximum value as per above comments - Correction of description for 'Direction of profile' (0-22-056)
amilan17 commented 2 years ago

https://github.com/wmo-im/CCT/wiki/Teleconference-1-and-2.11.2022 notes:

vturpin commented 2 years ago

Hi Anna, Thanks for this deadline. Very usefull ! Is there anyone in particular in charge of this validation results? I see no name in the meeting note shared by Dave.

amilan17 commented 2 years ago

https://github.com/wmo-im/CCT/wiki/Teleconference-21.11.2022 note: branch needs updating and needs samples @marianmajan-ibl, @SibylleK and @marijanacrepulja will validate

david-i-berry commented 2 years ago

I'm happy to help where I can but we really need some representative sample profiles. I had a quick look at the file linked by @justinbuck , this contains the parameters we are adding but contains far too much data (~500MB of data) to easily create a sample file. If someone is able to extract a few profiles, ensuring that the PHASE and PHASE_NUMBER variables are set I should be able to generate the test BUFR files for validation.

@vturpin @kevin-obrien @castelao

kbailey-noaa commented 2 years ago

@david-i-berry et al - Does the new BUFR template require PHASE and PHASE_NUMBER? The US Glider DAC file format is trajectory profile-based, so doesn't include those variables. Would a .nc file from that DAC be useful, despite not including those phase variables?

david-i-berry commented 2 years ago

Within the BUFR template there will be one dive per BUFR record and so there needs to be a way to determine and encode the phase of the profile and the individual profiles / dives from the data, including the start and end of each dive.

As part of the validation process we need examples with the new proposed Table B elements in order to test and ensure that we have the ranges and numerical precision correct.

vturpin commented 2 years ago

@kbailey-noaa, yes this would be usefull I think. Trajectory profile is a series of profile so I assume with such a data set, @david-i-berry could test the BUFR template. I don't think the BUFR template require the PHASE and PHASE_NUMBER. If I understand well the problem, the data set from BODC is a trajectory, so to extract the profile from this, you need the phase and phase_number completed. With a trajectory profile, like the US glider dac ones, there is no need for PHASE and PHASE_NUMBER any more.

kerfoot commented 2 years ago

@david-i-berry @amilan17 @vturpin I'm happy to provide sample data files, but am still unclear on exactly what type sample data files are required? We have lots of datasets available at the IOOS DAC for which profiles are available. The data sets are TrajectoryProfile, but I can provide sample queries to download individual (or groups) of profiles if they will suffice. Does this work?

castelao commented 2 years ago

@david-i-berry , are you looking for a BUFR-encoded file or just numbers for a dataset? Since this is for validation purposes only, would it make sense to use a synthetic dataset that explores some extreme values as well as cherry-picked resolution in certain ranges, even if that profile didn't make sense physically?

david-i-berry commented 1 year ago

@castelao I'm just after data to encode into the BUFR file (csv or netcdf would work). A synthetic dataset is good, especially if it explores the extreme values.

castelao commented 1 year ago

@vturpin @kerfoot , what are the sensors used for PAR and backscatter in the datasets that you handle? What are the nominal limits and resolutions for those sensors according to the manuals? That's all we need to create synthetic data to explore the extremes (absolute values and resolution). Even if such a 'profile' wouldn't represent a real-world structure, it should be able to accommodate actual measurements.

Once we get those reference values, it would be probably worth checking with someone publishing papers with those measurements if we are not missing anything.

vturpin commented 1 year ago

For PAR, the most frequent sensors used are : "Biospherical_instrument_PAR_2150 or 2155 they are similar (http://obsplatforms.plocan.eu/media/data_sheets/2021/05/QSP_QCP2150_SubmersiblePAR_Calibracion.pdf)" and "Satlantic_OCR504_ISCW" (file:///C:/Users/vturpin/Downloads/manual-OCR-504v2.pdf) . I found the manual but it is impossible for me to extract the min max values from the manual. For backscatter: "ECO_FLBBCD" - (https://www.seabird.com/eco-triplet/product?id=60762467720), chlorophyl range is "0–50 µg/L". But this is the CHLA range, not the output of the sensor...

@kerfoot could you extract min max from the PAR and backscatter variable from your ERDDAP? That would give us the extreme values I guess.

Note that those are the same sensors used on Argo. So we can use the same extreme values as they use for those variables.

Note also that those data can not be used in real time without metadata attached to it. So personnaly think we should not slow down the whole process of creating this BUFR format for variable that we don't know yet how to use it in real time.

It is good to have a framework to be able to push those data on the GTS, but it should not be a hurdle for validation. The BUFR format is expected by the glider community to progress on sharing data on GTS.

david-i-berry commented 1 year ago

Note also that those data can not be used in real time without metadata attached to it. So personnaly think we should not slow down the whole process of creating this BUFR format for variable that we don't know yet how to use it in real time.

I'd agree with this comment and have asked whether we can drop those new parameters (0-041-006 to 007, ChlA, backscatter and PAR) for now. If we can, we should be able to validate the sequence very quickly as it really is the min/max values etc for the new parameters that are holding up the validation. It also sounds like more metadata is required in the sequence, which would be another argument for not including them now.

Once it becomes clear what metadata we need to include (or what isn't currently included in the proposal and what needs to be added) we can submit a new proposal for the Table B elements and a new Table D sequence just for those data, similar to the approach used for Argo.

vturpin commented 1 year ago

It would be nice to have the opinion of the active people on this group. I agree to drop them in order to accelerate the validation and a creation of a dedicated BUFR format for gliders. @kerfoot @kbailey-noaa @justinbuck @castelao @gliderma and others, what do you think ? This work done will not be lost anyway.