BUFR Table for the variable Permafrost Temperature

rodicanitu commented 1 year ago

Details

Permafrost community (as supported through the Global Cryosphere Watch) is requesting the preparation of a BUFR Table for Permafrost Temperature, which is available from AWS operating in the Arctic, e.g. Svalbard Integrated Arctic Earth Observing System (SIOS. Topic lead: Dr Ketil Isaksen, ketili@met.no
Additional description to be provided by Ketil

Proposal

update 11 July 2023

Add sequence 3 07 064 (Soil temperature profile including permafrost)

FXY2	ElementName_en	ElementDescription_en
004025	Time period or displacement
106000	Delayed replication of 6 descriptors
031001	Delayed descriptor replication factor
007061	Depth below land surface
201131	Change data width
202129	Change scale
012130	Soil temperature
201000	Change data width	Cancel
202000	Change scale	Cancel

Note: "Intended for recording permafrost, but can be applied for any soil temperature profile"

Requestor

Ketil Isaksen (@KetilIsaksen), Norwegian Meteorological Institute, ketili@met.no
Rodica Nitu (@rodicanitu), WMO/GCW

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.25.26.April.2023 notes:

Anna will send an email to establish coordination

KetilIsaksen commented 1 year ago

I prepared some slides on permafrost measurements as a background for the discussion regarding what is needed from possible changes in BUFR tables and in order to hopefully reach some conclusions. Permafrost Temperature in BUFR.pdf

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.2&3May.2023 notes: Ketil presented an overview Next steps:

list all of the things needed to describe permafrost
meet to discuss and determine if current BUFR templates and codes will suffice
will need sample data in BUFR for validation

@amilan17 will organize a side meeting with Marijana, Antoine, Sibylle, Jeff, Ketil and Pal.

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.16.May.2023.permafrost notes: Pal described the findings from creating a sample data file in the email sent on 12 May. (see copy of email below).
DECISIONS:

Create a specified sequence for permafrost/soil temperature and depth will help data providers
start soil temperature sequence (?) with time period w/o a recommendation for a particular number (provider can decide)
Find a name to be inclusive of more generic without losing the main purpose for permafrost
- ~maybe use name similar to <<3 07 063>> Depth below land surface and soil temperature~
- Soil temperature profile including permafrost

NEXT STEPS:

@marijanacrepulja will update the issue summary with proposal and update branch
Validate the branch
Sample data with this sequence at the end for validation

amilan17 commented 1 year ago

email copy circa 12 May:

Dear colleagues,

We were asked to provide a sample of how permafrost temperature could be encoded in BUFR, and for this we have chosen the data set from our northernmost permafrost station 01002 Verlegenhuken of 06 UTC May 4th.

Please find attached the files:

wmo_1002-040600.bufr: This is the BUFR file produced internally at MET Norway for 01002 Verlegenhuken, and later included as subset 1 in the GTS bulletin ISMD38 ENMI 040600. Note that we are using the BUFR template (unexpanded descriptors in section 3) 307079 004025 011042 for all our land based stations, whether or not they are actually observing 011042 Maximum wind speed (10-minute mean wind).

wmo_1002-040600_permafrost.bufr: Here we have tried to encode also the 27 levels of permafrost temperature observed at Verlegenhuken, by adding the sequence 004025 104000 031001 202126 007061 202000 012130 to the set of unexpanded descriptors in section 3.

wmo_1002-040600.txt and wmo_1002-040600_permafrost.txt: Decoded contents of the two bufr files by use of bufrread.pl from (the CPAN module) Geo::BUFR.

Explanation of the added descriptor sequence in the permafrost BUFR file:

004025 Time period or displacement: Set to 0 106000 Delayed replication of 6 descriptors 031001 Delayed descriptor replication factor 007061 Depth below land surface 201131 Change data width: Add 3 bits 202129 Change scale operator: Add 1 to scale 012130 Soil temperature 202000 Change scale: Cancel 201000 Change data width: Cancel

Change of data width and scale for 012130 Soil temperature is regarded as necessary to obtain 3 decimal precision in temperature. Although the measurement systems currently in use generally provide an accuracy and precision of 0.1°C or better, 3 decimal in the data series is our preferred precision to better capture small, but significant long-term changes in permafrost temperature.

The first descriptor 004025 (set to 0) is not really part of the permafrost sequence, but necessary here to mark transition from 'period data section' to a new 'instantaneous data section' in accordance with e.g.

B/C1.9 “Instantaneous” data required by regional or national reporting practices: If regional or national reporting practices require inclusion of additional “instantaneous” parameters, the sequence descriptor 3 07 080 shall be supplemented by the required element descriptors being preceded by a relevant time period descriptor set to zero, i.e. 0 04 024 = 0 or 0 04 025 = 0.

It is our current point of view that 012130 Soil temperature very well could be used also for permafrost temperature. There should be no need to include more metadata about permafrost temperature in the BUFR file we believe, better then to add such metadata in Oscar Surface for the station in question.

wmo_1002-040600_permafrost.txt wmo_1002-040600.txt permafrostBUFR.zip

marijanacrepulja commented 1 year ago

	Soil temperature profile including permafrost
004025	Time period or displacement
106000	Delayed replication of 6 descriptors
031001	Delayed descriptor replication factor
007061	Depth below land surface
201131	Change data width
202129	Change scale operator
012130	Soil temperature
202000	Change scale	Cancel
201000	Change data width	Cancel

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.6.7.June.2023 notes: Marijana updated the branch and samples are available; @sergioh-pessoal can validate; Team would like to better name, for example:

Soil temperature profile (intended for permafrost)
Soil temperature profile, such as for permafrost

sergioh-pessoal commented 1 year ago

I decoded the available sample. The result is attached and is the same as the provided text. Version 31 was used for coding this sample. Therefore, the descriptor 3-07-064 was not used. I would like to ask. Does a new sample need to be produced? Including the new descriptor wmo_1002-040600_permafrost.bufr.decoded.txt.gz

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.6.7.June.2023 notes:

DECISION on title: Keep current title "Soil temperature profile including permafrost" and add a comment indicating that it's not limited to only permafrost;

note: "Intended for recording permafrost, but can be applied for any soil temperature profile"

amilan17 commented 1 year ago

samples and copy of email circa 06-June-2023

I have copied the ecCodes tables (at bufr/tables/0/wmo/31/) needed for decoding of the original BUFR sample (the file wmo_1002-040600_permafrost.bufr which was encoded with WMO master table 31) to the new location bufr/tables/0/wmo/40/, and then inserted the new sequence descriptor 307064 in sequence.def as

"307064" = [ 004025, 106000, 031001, 007061, 201131, 202129, 012130, 202000, 201000 ]

The resulting BUFR file and decoded contents are attached as wmo_1002-040600_permafrost_307064.bufr and wmo_1002-040600_permafrost_307064.txt

As you will see from the last file, the unexpanded sequence of descriptors in section 3 is now 307079 004025 011042 307064.

However, as there was some debate at the previous meeting whether or not to include 004025 "Time period or displacement" in the new sequence descriptor, I have for convenience also attached files where 004025 is omitted from 307064. Therefore the unexpanded sequence of descriptors in section 3 in wmo_1002-040600_permafrost_307064_without_004025.txt is 307079 004025 011042 004025 307064.

Note that there will be minor differences in the files of decoded contents from our last email due to this time using ecCodes tables instead of BUFRDC tables for the decoding. sampleData.zip

jbathegit commented 1 year ago

I was able to decode both samples (with and without 004025) using NCEP's BUFR decoding software, and in both cases I got identical results to what was provided in the corresponding .txt files. So I believe that makes for a successful validation, but note that this new 307064 sequence will eventually be part of version 41 of the tables, not version 40 which is already released.

As far as whether or not to include 004025 in the 307064 sequence, I don't have a strong opinion either way, but my personal vote would be to include it.

sergioh-pessoal commented 1 year ago

I also decoded both samples (with and without 004025) and get the same results using INPE BUFR decoding.
Regarding the inclusion or not of 004025 in the 307079, I don’t have opinion yet, but my first impression is that it should be included

amilan17 commented 1 year ago

All please review the issue summary.

I updated it based on content in the branch. @marijanacrepulja there are some descriptions in this comment (https://github.com/wmo-im/BUFR4/issues/157#issuecomment-1576967231) that are not in the branch.

sergioh-pessoal commented 1 year ago

@amilan17

I double-checked the table in the branch. And I decoded the sample again using the branch tables. The sample (with 0-04-025 included) was decoded without any problems and the result has exactly the same values as the provided text file

Reviewing the question of whether or not 0-04-025 should be included in 3-07-064. The best option is the one already in the tables. i.e. The 0-04-025 included in 3-07-064.

Reason: Ensures time period = 0 is being set

marijanacrepulja commented 1 year ago

All please review the issue summary.

I updated it based on content in the branch. @marijanacrepulja there are some descriptions in this comment (#157 (comment)) that are not in the branch.

@amilan17 I updated proposal to avoid confusion, the comments were just for info. Thanks for spotting this.

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.13.July.2023 notes:

note is not needed (as requested in PR); branch is ready;

wmo-im / BUFR4