search

wmo-im / BUFR4

BUFR edition 4
MIT License
27 stars 9 forks source link

Table D, proposal for an new BUFR template dedicated to drifting balloon platforms: 3 11 013 #208

Open adoerenbecher opened 1 month ago

adoerenbecher commented 1 month ago

Initial request

Rationale: Field campaigns with long duration drifting balloons are quite common/frequent. Sometimes high-value data are available in real-time and could be used in numerical weather prediction applications (e.g. data assimilation).

Drifting balloons are a world unto themselves and a very wide variety of measurements can be collected using them (both in-situ and remotely sensed data, e.g. radio-occultation) because these are primarily vectors. These vectors are chosen for their ability to transport sensors to atmospheric regions that would otherwise be difficult to access.

These drifting systems are very often fitted with sensors to measure the close environment of the balloon, and these are the data of interest here.

With this issue we propose to set up a new BUFR template to allow the dissemination of in-situ data (at the flight level) to the meteorological community. Such a BUFR template would be a trade-off between aircraft or radiosonde messages, involving the definition of new tables as a few novel concepts should be defined.

Purpose: The requestor (Météo-France Research Department, aka CNRM) has some experience in using observations collected by balloons operated par CNES (French National Space Agency) in various field campaigns such as CONCORDIASI, Stratéole-2 (stratospheric data in the global model) or HyMeX (marine boundary layer above the Mediterranean in the regional model). During these campaigns the data were made available using FTP repositories. Having an official BUFR template dedicated to these balloons data would also help in the dissemination of the data to the community.

The Stratéole-2 is not over and a last (third) campaign will take place at the end of 2026 with flights circumnavigating the tropical tropopause layer (TTL) for months with a flotilla of about 20 balloons drifting at either 18 or 20km of altitude. Many experimental systems will collect research data, but the TSEN system (IPSL/LMD, France) is able to collect high quality data and send them in real-time through satellite link. Météo-France participates in the Stratéole-2 effort and is ready to disseminate the data on the global telecommunication system if WMO allows it, and having a proper BUFR template is part of it.

At CNRM, we have designed a preliminary local version of a BUFR template to make tests with the data of the two first campaigns of the Stratéole-2 project. This dedicated BUFR format allow both the IT system and data assimilation systems to handle balloon data properly. For these data which are neither radiosonde, neither aircraft data, it is crucial to properly control black or whitelisting, observation uncertainties or thinning procedures.

Implementation plan: Here we simply submit our attempts to build a sensible template (3 11 012) to your expertise. We have used new code for descriptors corresponding to the quantities that were not already defined in the BUFR documentation. We hope that these do not clash with recent descriptors. Depending on the decisions of the WMO experts, we can adjust the preliminary template we have build and provide WMO with a sample of real data collected during the past Stratéole-2 data. As soon as the BUFR template is validated we will ask WMO for an authorization to disseminate the Stratéole-2 data. If authorized, we will prepare the data flow in order to be ready in September 2026.

Amendment details

The tables below describe the descriptors we are proposing for the new BUFR message model for data from drifting balloons. We have tried to follow the format used in the official BUFR documentation. The last column gives examples of values for data from the Stratéole-2 campaign.

Most of the descriptors used here already exist.

However, new descriptors need to be created, in particular for metadata describing the balloons and also how the data is collected. In the case of Stratéole-2, the measurement station is not compact like a radiosonde and the measurement points are distributed vertically: it is therefore necessary to be able to describe this staggering in relation to the point where the altitude is measured (GNSS). We have put a ‘new’ label next to the descriptors to be created.

We include the necessary structure for humidity information, even though Stratéole-2/TSEN data do not have humidity. But in case it may happen with other balloons (e.g. encoding for Windborne data) the structure is already there.

The following section (publication) lists the descriptors to be created.

CATEGORY ELEMENT NAME TABLE REFERENCE UNIT SCALE REF. VALUE DATA WIDTH STATUS COMMENTS EXAMPLE (case of TSEN Strateole-2)
WIGOS Wigos Identifier Series 0-01-125 Numeric 0 0 4 existing 0
WIGOS Wigos Issuer Of Identifier 0-01-126 Numeric 0 0 16 existing 250 (France in ISO-3166)
WIGOS Wigos Issue Number 0-01-127 Numeric 0 0 16 existing 0
WIGOS Wigos Local Identifier Character 0-01-128 CCITT IA4 0 0 128 existing "STR2C201"
Carrier Balloon flight number 0-01-156 (new) CCITT IA4 0 0 64 to be proposed curr. vacant "STR2C201"
Carrier Balloon Registration number or other identification 0-01-157 (new) CCITT IA4 0 0 64 to be proposed curr. vacant "Stratéole-2"
Carrier Originating centre 0-01-035 Common Code Table C-11 0 0 8 existing 85 (Météo-France, Toulouse)
Carrier Identification of originating/generating sub-centre 0-01-034 Common Code Table C-12 0 0 16 to be proposed curr. vacant 205 (CNES)
Carrier Type of balloon envelope 0-01-158 (new) Code Table 0 0 16 to be proposed curr. vacant 0 (SPB)
Carrier Typical size of balloon enveloppe 0-01-159 (new) m 0 0 12 existing 12 (ballon diameter)
Carrier Phase of balloon flight 0-08-100 (new) Code Table 0 0 4 to be proposed curr. vacant ascent, ceiling, descent, drop, unsteady
Sonde Type of station 0-02-001 Code Table 0 0 2 existing 0 (Automatic)
Sonde Radiosonde type 0-02-011 Code Table 0 0 8 to be proposed curr. vacant 136 (TSEN)
Sonde Precision of temperature observation 0-02-005 K 2 0 7 existing 0.05 (values for TSEN)
Sonde Precision of pressure observation 0-02-192 (new) Pa -1 0 14 to be proposed curr. vacant 10 (values for TSEN)
Sonde Precision of mixing ratio observation 0-02-193 (new) kg/kg -1 0 14 to be proposed curr. vacant missing (no value for TSEN)
Sonde Tracking technique/status of system used 0-02-014 Code Table 0 0 7 existing 8 (Automatic satellite navigation)
Sonde Type of measuring equipment used 0-02-003 Code Table 0 0 4 existing 7 (satellite navigation)
Sonde pressureSensorType 0-02-095 Code Table 0 0 5 existing 30 (other: piezo-electric transducer)
Sonde temperatureSensorType 0-02-096 Code Table 0 0 5 existing 1 (bead thermistor)
Sonde humiditySensorType 0-02-097 Code Table 0 0 5 existing 31 (missing)
Sonde Sample averaging period 0-25-171 s 0 0 10 existing 900
Sonde Sample size 0-25-172 (?) Numeric 0 0 10 existing 30
Date/Time Time significance 0-08-021 Code Table 0 0 6 existing 26 (Time of last known position)
Date/Time Year 0-04-001 a 0 0 12 existing 2018
Date/Time Month 0-04-002 mon 0 0 4 existing 12
Date/Time Day 0-04-003 d 0 0 6 existing 1
Date/Time Hour 0-04-004 h 0 0 5 existing 0
Date/Time Minute 0-04-005 min 0 0 6 existing 0
Date/Time Second 0-04-006 s 0 0 6 existing 0
Positionning Latitude 0-05-001 degree 5 -9000000 25 existing high accuracy
Positionning Longitude 0-06-001 degree 5 -18000000 26 existing high accuracy
Positionning Height 0-07-007 m 0 -1000 17 existing at GPS rec.
Positionning Station elevation quality mark (for mobile stations) 0-33-024 Code Table 0 0 4 existing 2 or 1 (values for TSEN system)

Here follow the descriptors for the data :

CATEGORY ELEMENT NAME TABLE REFRENCE UNIT SCALE REF. VALUE DATA WIDTH STATUS COMMENTS EXAMPLE
Measurements Pressure 0-07-004 Pa -1 0 14 existing
Measurements Height of pressure sensor wrt station 0-07-077 (new) m 0 -400 12 to be proposed 0
Measurements Geopotential height 0-10-009 gpm 0 -1000 17 existing
Measurements Temperature/air temperature 0-12-001 K 1 0 12 existing
Measurements Height of temperature sensor wrt station 0-07-079 (new) m 0 -400 12 to be proposed -10
Measurements Standard deviation of temperature 0-12-051 K 0 0 10 existing
Measurements Water vapour mixing ratio 0-13-002 kg/kg 5 0 14 existing missing
Measurements Height of humidity sensor wrt station 0-07-080 (new) m 0 -400 12 to be proposed missing
Measurements Standard deviation of mixing ratio 0-13-165 (new) kg/kg 5 0 14 to be proposed curr. vacant missing
Measurements Wind direction 0-11-001 degree true 0 0 9 existing
Measurements Wind speed 0-11-002 m/s 1 0 12 existing
Measurements Height of wind sensor wrt station 0-07-078 (new) m 0 -400 12 to be proposed 10
Measurements Standard deviation of wind direction 0-11-049 degree true 0 0 9 existing
Measurements Standard deviation of wind speed 0-11-050 m/s 1 0 12 existing

As we introduce new Code tables... Here are the information for the code figures to be used.

Descriptor 0-01-158 for "Type of balloon envelope" Code figure Meaning
0 Reserved
1 SPB (Super-Pressurized Balloon) without altitude control
2 SPB (Super-Pressurized Balloon) with altitude control
3 ZPB (Zero-Pressure Balloon)
4 IRM (Infra-Red Montgolfier)
5 Airship (Airship-shaped Balloon)
6 Tetrodon (NOAA tetraedric Balloon)
15 Missing value
Descriptor 0-02-011 for "Sonde type" : add a new entry Code figure Meaning
136 TSEN
Descriptor 0-08-100 for "Phase of balloon flight" Code figure Meaning
0 Reserved
1 Ascent
2 Ceiling
3 Descent
4 Drop (falling station)
5 Unsteady (during manoeuvres)
15 Missing value

Comments

We have a PDF document about this BUFR template proposal. It is available on demand.

Requestor(s)

DOERENBECHER Alexis, Météo-France, Reaserche Center (CNRM) and Stratéole-2 project (alex.doerenbecher@meteo.fr) POURRET Vivien, Météo-France, Reaserche Center (CNRM) and Stratéole-2 project (vivien.pourret@meteo.fr)

Stakeholder(s)

The amendment has been designed to not change (hopefully) the existing BUFR matter, but only to add new features to the BUFR system. All the research and operational numerical weather prediction centers might be impacted as soon as they try to decode balloon data that would have been disseminated by the Météo-France/Stratéole-2 project on the global telecommunication system by the end of 2026 (see "initial request" section for more details on the implementation plan).

Publication(s)

Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Table D, Class/Category 11, add BUFR descriptor 3 11 013 (template) Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 01, code table 0 02 011: add TSEN station Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 01, add BUFR descriptor 0 01 156 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 01, add BUFR descriptor 0 01 157 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 01, add BUFR descriptor 0 01 158 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 01, add BUFR descriptor 0 01 159 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 02, add BUFR descriptor 0 02 192 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 02, add BUFR descriptor 0 02 193 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 07, add BUFR descriptor 0 07 077 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 07, add BUFR descriptor 0 07 078 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 07, add BUFR descriptor 0 07 079 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 07, add BUFR descriptor 0 07 080 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 08, add BUFR descriptor 0 08 100 Manual on Codes (WMO-No. 306), Volume I.2, BUFR, Class 13, add BUFR descriptor 0 13 165

Expected impact of change

MEDIUM

Collaborators

Collaborators are the principal investigators (PI) in the Stratéole-2 project. Stratéole-2 project is a French-American project to better understand atmospheric mechanisms at stake in the Earth climate system, focusing on the Tropical Tropopause Layer and hydration processes in relation with deep convection. Stratéole-2 has 2 field campaigns involving instrumented stratospheric balloons operated by CNES (plus a test campaign). Scientific payloads may vary from one balloon to another and most of these are cutting edge systems to collect unprecedented dataset. However, all balloons are fitted with high quality instrumentation to measure pressure and temperature at flight level and also estimate horizontal wind (TSEN system, IPSL/LMD). The data collect by the latter are available at high-frequency and in near-real-time. These data are of high value global numerical weather prediction activities. All other data are available on open database after the campaigns. The PI for the TSEN system is Albert Hertzog (IPSL/LMD). He is responsible for the raw data collection and preprocessing. The CNES chief manager of the balloon operations is Stephanie Venel.

References

Here follow a few references about Stratéole-2. TSEN website: https://www.lmd.ipsl.fr/en/instruments-lmd/tsen-2/ Stratéole-2 in EOS: https://eos.org/science-updates/around-the-world-in-84-days Stratéole-2 white paper: https://www.lmd.polytechnique.fr/VORCORE/Documents/US%20Involvement%20in%20the%20Strateole%202%20Balloon%20Campaign.pdf

Validation

A prototype of this BUFR template for balloons has been built at Météo-France (research center) on the basis of local tables that can be used with EcCodes.

Data assimilation experiments in the ARPEGE global model have thus been carried out and the impact of TSEN data from the 2 Stratéole-2 campaigns that have already taken place (test campaign and campaign n°1) has been measured. A scientific article on the results is being prepared and is not yet ready for distribution.

The results were presented at the Stratéole-2 conference in Boulder (USA) in June 2023. The presentation is available on request.

adoerenbecher commented 1 month ago

Sorry, I've forgotten to include a diagram that shows how the TSEN system is built - especially the vertical distance between the sensors. Here is the picture. !

strateole_balloon_parameters

amilan17 commented 1 month ago

https://github.com/wmo-im/tt-tdcf/wiki/2024.10.16.tt.tdcf notes:

adoerenbecher commented 1 week ago

Following the meeting on Oct. 16, I formalised the sequence of this proposed template. The description above is more or less a list of content. Colleagues explained to me how e well-defined sequence limits the number of new entries/descriptors to be created.

For example, it is not necessary to define a height in relation to the station for each parameter/sensor; it is sufficient to use a generic descriptor and to use the sequence "pressure", "temperature", "humidity" and "winds" to specify the vertical positioning of each sensor under the balloon.

The same logic applies to the sampling description. In the case of Stratéole 2, some raw data are acquired at high frequency and are averaged subsequently. The descriptors 0-25-170 and 0-25-171 are used to give this information. It is updated for each sensor, as each sensor may have its own acquisition protocol.

As a consequence, the measurement section should better read as follows.

CATEGORY ELEMENT NAME TABLE REFRENCE UNIT SCALE REF. VALUE DATA WIDTH STATUS COMMENTS EXAMPLE
Time significance 0-08-021 Code Table 0 0 6 existing 2
Height of sensor wrt station 0-07-077 (new) m 0 -400 12 to be proposed 0
Sampling interval 0-25-170 s 0 0 10 existing 2
Sample averaging period 0-25-171 s 0 0 10 existing 600
Pressure 0-07-004 Pa -1 0 14 existing
Geopotential height 0-10-009 gpm 0 -1000 17 existing
Height of sensor wrt station 0-07-077 (new) m 0 -400 12 to be proposed -10
Sampling interval 0-25-170 s 0 0 10 existing 2
Sample averaging period 0-25-171 s 0 0 10 existing 600
Temperature/air temperature 0-12-001 K 1 0 12 existing
Standard deviation of temperature 0-12-051 K 0 0 10 existing
Height of sensor wrt station 0-07-077 (new) m 0 -400 12 to be proposed missing
Sampling interval 0-25-170 s 0 0 10 existing missing
Sample averaging period 0-25-171 s 0 0 10 existing missing
Water vapour mixing ratio 0-13-002 kg/kg 5 0 14 existing missing
Standard deviation of mixing ratio 0-13-165 (new) kg/kg 5 0 14 to be proposed curr. vacant missing
Height of sensor wrt station 0-07-077 (new) m 0 -400 12 to be proposed 10
Sampling interval 0-25-170 s 0 0 10 existing 60
Sample averaging period 0-25-171 s 0 0 10 existing 600
Wind direction 0-11-001 degree true 0 0 9 existing
Wind speed 0-11-002 m/s 1 0 12 existing
Standard deviation of wind direction 0-11-049 degree true 0 0 9 existing
Standard deviation of wind speed 0-11-050 m/s 1 0 12 existing

The template is open to the introduction of other parameters in relation with e.g. atmospheric chemistry.

adoerenbecher commented 1 week ago

I attach here a PDF document to better show the clarified sequence for this template.

BUFR_proposal_sequence_v2.pdf

jbathegit commented 1 week ago

As noted by a colleague within DMI, the data width for the proposed new 0-01-158 ("Type of balloon envelope") descriptor appears to be incorrect. It is listed as 16, when a look at the corresponding proposed code table indicates the data width should be 4 instead.

amilan17 commented 5 days ago

https://github.com/wmo-im/tt-tdcf/wiki/2024.11.12.tt.tdcf notes: