Closed efucile closed 2 years ago
amilan17
commented
3 years ago BUFR file from @marijanacrepulja
https://wmoomm.sharepoint.com/:u:/s/wmocpdb/EXQYVVfNGAJOi21VRMNpZ6kBjiNfD9eGoUOE8F08bRg5Xg?e=I5ItAo
posted copy on SharePoint.... Is this corrupted too?
marijanacrepulja
commented
3 years ago @amilan17 BUFR file looks good now, thanks!
jitsukoh
commented
3 years ago Just a reminder for tomorrow's discussion, existing table B descriptors are Depth of fresh snow (0 13 012) and Total snow depth (0 13 013). BUFR tables were developed before the WIGOS Metadata and the vocabulary is not consistent between the two. We sometimes change the element name of existing table B descriptors but I am not sure if we should do this for these descriptors, as they have been used in SYNOP for long time and could cause confusion among users... Would it be an idea to change to something like 0 13 012 Snow depth (Total snow depth) to indicate another name?
jitsukoh
commented
3 years ago @DenisStuber @sergioh-pessoal @marijanacrepulja Here is the updated proposal for the new DAYCLI message based on today's discussion. Please let me know if you find any errors/misunderstanding...
1. Add a new entry in the significance qualifier class of table B (0-08-094) and a new code table to describe the method used to calculate the daily average temperature.
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-094 | Method used to calculate the average daily temperature | Code table | 0 | 0 | 5 |
Discussion: See above.
CODE TABLE 0-08-094 0-08-094 Method used to calculate the average daily temperature
| Code Figure | |
|---|---|
| 0 | Average of maximum and minimum values: Tm = ( Tx + Tn) / 2 |
| 1 | Average of the 8 tri-hourly observation |
| 2 | Average of 24 hourly observation |
| 3 | Weighted average of 3 observations: Tm = (aT1 +bT2 + cT3), |
| 4 | Weighted average of 3 observation and also maximum and minimum values: Tm= (aT1 +bT2 + cT3 +dTx + eTn) |
| 5 | AWS complete integration from minute data |
| 6 -30 | Reserved |
| 31 | Missing value |
Note : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x ”And“ n represent the values measured at different times or maximum (x) or minimum (n) values.
2. Quality flag the for each climatological value (Temperatures, Precipitation and Snow). Add a Associated filed with xx bits and update the Associated Field Significance (0 31 021) : Code figure 10: xx bit indicator for the Quality Flag for climatological values.
Action item for SC-CLI/ET-DRC The necessary range (number of bits) and the flagging system will be prepared by SC-CLI/ET-DRC.
3. Siting classification and Measurement Quality Classification
Discussion: It is necessary to add the Siting Classification (SC) and also Measurement Quality Classification (MQC). However, the MQC must be accompanied by the siting classification.
Proposal: Add 2 entries in table B: One for Temperature and another for Precipitation. Both with 2-character (16 bits), where, the first character represents the SC Class from "1" to "5", as defined in the CIMO Guide (WMO No.8) Part I, Chapter I, Annex 1B: and the second character, represent class “A” to “E”. (CIMO Guide)
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-095 | Siting and Measurement Quality Classification for temperature | CCITTIA5 | 0 | 0 | 16 |
| 0-08-096 | Siting and Measurement Quality Classification for precipitation | CCITTIA5 | 0 | 0 | 16 |
Note: It is good to have these metadata in each message instead of having them only in OSCAR/Surface. Note: At this moment, there is not requirement for this kind of descriptor for snow.
4. Element names for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) Update the element name for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) to add terminology used in WIGOS Metadata.
| F X Y | Current | Updated (proposal) |
|---|---|---|
| 0 13 012 | Depth of fresh snow | Depth of fresh snow (Depth of snowfall) |
| 0 13 013 | Total snow depth | Total snow depth (Snow Depth) |
5. The new DAYCLI BUFR message
Update descriptor 3-07-YYY (Update the sequence or create a new descriptor in the next version of BUFR table D)
| Table reference | Table references | Element name | Description / comments |
|---|---|---|---|
| 3-07-YYY | Supplemental daily temperature and precipitation values for monthly climate report | ||
| 3-01-150 | WIGOS identifier | ||
| 3-01-001 | WMO block and station number | ||
| 3-01-021 | Latitude/longitude (high accuracy) | ||
| 0-07-030 | Height of station ground above mean sea level (see note 3) (m) | ||
| 0-08-095 | Siting and measurement quality classification for temperature | Character- Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-096 | Siting and measurement quality classification for precipitation | Character- Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-094 | Method used to calculate the average daily temperature | (code table) | |
| 3-01-001 | Year/Month/Day | ||
| (Total accumulated precipitation) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-0xx | Add associated field | xx bits long | |
| 0-31-021 | Associated Field Significance | Set as 10 for indicator for the Quality Flag for climatological values | |
| 0-13-060 | Total accumulated precipitation (kg m-2) | ||
| 2-04-000 | Add associated field | cancel | |
| (Depth of fresh snow) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-0xx | Add associated field | xx bits long | |
| 0-31-021 | Associated Field Significance | Set as 10 for indicator for the Quality Flag for climatological values | |
| 0-13-012 | Depth of fresh snow (Depth of snowfall) | ||
| 2-04-000 | Add associated field | cancel | |
| (Total snow depth) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-0xx | Add associated field | xx bits long | |
| 0-31-021 | Associated Field Significance | Set as 10 for indicator for the Quality Flag for climatological values | |
| 0-13-013 | Total snow depth (Snow depth) | ||
| 2-04-000 | Add associated field | cancel | |
| (Max, Min, Mean temperature) | |||
| 0-07-032 | Height of sensor above local ground | for temperature measurement | |
| 1-07-003 | Replicate 7 descriptors 3 times | ||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 0-08-023 | First-order statistics | 2 – maximum; 3 – minimum; 4 – mean | |
| 2-04-0xx | Add associated field | xx bits long | |
| 0-31-021 | Associated Field Significance | Set as 10 for indicator for the Quality Flag for climatological values | |
| 0-12-101 | Temperature/air temperature (k) | ||
| 2-04-000 | Add associated field | cancel | |
| 0-08-023 | First-order statistics (code table) | Set as missing value |
Action item for Sergio Produce samples using the above sequence to see if real practices can be represented.
6. Use of subset In this message, each subset is used to represent data for each day. Normally each subset is used to represent data from each station. Some guidance will be necessary to standardize how to encode messages from different stations.
jitsukoh
commented
3 years ago @amilan17 can you tell me what to use as YYY for this new sequence (3-07-YYY)?
amilan17
commented
3 years ago I think it can be 3-07-104, 3-07-183 OR 3-07-075.
@efucile Can you provide insight?
The sequences in the 3-07-07* range seem to be a closer fit.
| 307071 | (Monthly values of a land station) |
| 307072 | (Monthly normals for a land station) |
| 307073 | (Representation of CLIMAT data of the actual month and for monthly normals) |
| 307074 | (Supplemental daily temperature and precipitation values for monthly climate report) |
| 307075 | |
| 307076 | (Monthly values from a land station in compliance with regional or national reporting practices) |
| 307077 | (Monthly normals for a land station in compliance with regional or national reporting practices) |
| 307078 | (Sequence for representation of monthly values suitable for CLIMAT data in compliance with regional or national reporting practices) |
| 307079 | (Sequence for representation of synoptic reports from fixed land stations suitable for SYNOP data and for maritime data from coastal stations) |
These are the last two sequences in Table D07. I don't know why it jumped from 103 to 182.
| 07 | Surface report sequences (land) | 307103 | (Snow observation, snow density, snow water equivalent) |
| 07 | Surface report sequences (land) | 307182 | (Sequence for representation of synoptic reports from a fixed land station suitable for SYNOP data in compliance with reporting practices in RA II) |
jitsukoh
commented
3 years ago Thank you @amilan17 it seems 3 07 075 is the best fit. I take it for now and if there is anything that we need to consider we'll see again.
sergioh-pessoal
commented
3 years ago @jitsukoh and @marijanacrepulja Here is a sample of the Chardonix station data encoded with the latest template and using the proposed descriptor 3-07-075. Note that this is only a preliminary sample. The quality control part needs more work. Chamonix_202012_307075.bufr.gz
jitsukoh
commented
3 years ago @sergioh-pessoal @marijanacrepulja I updated the proposal for the new DAYCLI message, including the QC result using an associated field of 4-bit indicator. Please let me know how you feel.
1. Add a new entry in the significance qualifier class of table B (0-08-094) and a new code table to describe the method used to calculate the daily average temperature.
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-094 | Method used to calculate the average daily temperature | Code table | 0 | 0 | 5 |
Discussion: See above.
CODE TABLE 0-08-094 0-08-094 Method used to calculate the average daily temperature
| Code Figure | |
|---|---|
| 0 | Average of maximum and minimum values: Tm = ( Tx + Tn) / 2 |
| 1 | Average of the 8 tri-hourly observation |
| 2 | Average of 24 hourly observation |
| 3 | Weighted average of 3 observations: Tm = (aT1 +bT2 + cT3), |
| 4 | Weighted average of 3 observation and also maximum and minimum values: Tm= (aT1 +bT2 + cT3 +dTx + eTn) |
| 5 | AWS complete integration from minute data |
| 6 -30 | Reserved |
| 31 | Missing value |
Note : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x ”And“ n represent the values measured at different times or maximum (x) or minimum (n) values.
2. Quality flag for each climatological value (Temperatures, Precipitation and Snow). Add a new entry (5) of 4-bit indicator of quality control in the Associated field significance (0 31 021) (code table is tentative) (the code figure was changed from 10 to 5 because 5 seems a closer fit.)
| Code Figure | Code table | |
|---|---|---|
| 3 - 4 | reserved | |
| 5 | 4-bit indicator of quality control | 0 = Not controlled, 1 = Controlled and correct, 2 = Controlled and suspect, 3-14 = Reserved, 15 = Missing value |
Action item for SC-CLI/ET-DRC The code table will be finalized by SC-CLI/ET-DRC.
3. Sitting classification and Measurement Quality Classification
Discussion: It is necessary to add the Sitting Classification (SC) and also Measurement Quality Classification (MQC). However, the MQC must be accompanied by the sitting classification.
Proposal: Add 2 entries in table B: One for Temperature and another for Precipitation. Both with 2-character (16 bits), where, the first character represents the SC Class from "1" to "5", as defined in the CIMO Guide (WMO No.8) Part I, Chapter I, Annex 1B: and the second character, represent class “A” to “E”. (CIMO Guide)
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-095 | Sitting and Measurement Quality Classification for temperature | CCITTIA5 | 0 | 0 | 16 |
| 0-08-096 | Sitting and Measurement Quality Classification for precipitation | CCITTIA5 | 0 | 0 | 16 |
Note: It is good to have these metadata in each message instead of having them only in OSCAR/Surface. Note: At this moment, there is not requirement for this kind of descriptor for snow.
4. Element names for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) Update the element name for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) to add terminology used in WIGOS Metadata.
| F X Y | Current | Updated (proposal) |
|---|---|---|
| 0 13 012 | Depth of fresh snow | Depth of fresh snow (Depth of snowfall) |
| 0 13 013 | Total snow depth | Total snow depth (Snow Depth) |
5. The new DAYCLI BUFR message
Update descriptor 3-07-075 (Update the sequence or create a new descriptor in the next version of BUFR table D)
| Table reference | Table references | Element name | Description / comments |
|---|---|---|---|
| 3-07-075 | Supplemental daily temperature and precipitation values for monthly climate report | ||
| 3-01-150 | WIGOS identifier | ||
| 3-01-001 | WMO block and station number | ||
| 3-01-021 | Latitude/longitude (high accuracy) | ||
| 0-07-030 | Height of station ground above mean sea level (see note 3) (m) | ||
| 0-08-095 | Sitting and measurement quality classification for temperature | Character- Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-096 | Sitting and measurement quality classification for precipitation | Character- Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-094 | Method used to calculate the average daily temperature | (code table) | |
| 3-01-001 | Year/Month/Day | ||
| (Total accumulated precipitation) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-004 | Add associated field | 4 bits long | |
| 0-31-021 | Associated Field Significance | Set as 5 for 4-bit indicator of quality control | |
| 0-13-060 | Total accumulated precipitation (kg m-2) | ||
| 2-04-000 | Add associated field | cancel | |
| (Depth of fresh snow) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-004 | Add associated field | 4 bits long | |
| 0-31-021 | Associated Field Significance | Set as 5 for 4-bit indicator of quality control | |
| 0-13-012 | Depth of fresh snow (Depth of snowfall) | ||
| 2-04-000 | Add associated field | cancel | |
| (Total snow depth) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-004 | Add associated field | 4 bits long | |
| 0-31-021 | Associated Field Significance | Set as 5 for 4-bit indicator of quality control | |
| 0-13-013 | Total snow depth (Snow depth) | ||
| 2-04-000 | Add associated field | cancel | |
| (Max, Min, Mean temperature) | |||
| 0-07-032 | Height of sensor above local ground | for temperature measurement | |
| 1-07-003 | Replicate 7 descriptors 3 times | ||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 0-08-023 | First-order statistics | 2 – maximum; 3 – minimum; 4 – mean | |
| 2-04-004 | Add associated field | 4 bits long | |
| 0-31-021 | Associated Field Significance | Set as 5 for 4-bit indicator of quality control | |
| 0-12-101 | Temperature/air temperature (k) | ||
| 2-04-000 | Add associated field | cancel | |
| 0-08-023 | First-order statistics (code table) | Set as missing value |
6. Use of subset In this message, each subset is used to represent data for each day. Normally each subset is used to represent data from each station. Some guidance will be necessary to standardize how to encode messages from different stations.
jitsukoh
commented
3 years ago ET-DRC and TT-TDCF agreed on the validation process with the schedule below:
marijanacrepulja
commented
3 years ago @jitsukoh The QC result using an associated field of 4-bit indicator looks good to me.
amilan17
commented
3 years ago per meeting discussion: in validation once the data width for 0-08-094 (Method used to calculate the average daily temperature) and representation of sitting classifications (code table solution) will be finalized, sample will be provided and validated by IBL and Met Office
jitsukoh
commented
3 years ago @DenisStuber TT-TDCF discussed the proposal at yesterday's meeting and has two suggestions.
1. 0-08-094 Method used to calculate the average daily temperature Now this code table has 32 figures (5 bits) but we think it is better to have more space, probably 8 bits to ensure the table is not filled.
2. Sitting classification This is rather a question about the nature of classification. According to decision 4.1.2/1 (INFCOM-1) classification is aligned with OSCAR/Requirements. Our understanding is that the table therefore can change over time, and if this is the case, the table used needs to be specified to interpret the information, i.e. there should be a table version control and the version needs to be specified in BUFR message; otherwise in the future, interpretation of this information of archive data would be impossible.
One proposal of solution to specify the table version in BUFR message would be to define a code table instead of encoding the classifications as two characters, and specify the table version used in the code table (and update the code table when the table version changes), such as
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-095 | Sitting and Measurement Quality Classification for temperature | Code table | 0 | 0 | 8 |
| 0-08-096 | Sitting and Measurement Quality Classification for precipitation | Code table | 0 | 0 | 8 |
| CODE TABLE for 0-08-095 and 0-08-096 Code Figure | |
|---|---|
| 0 | Reserved |
| 1 | 1A (Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 2 | 1B (Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 3 | 1C (Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 4 | 1D (Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 5 | 1E (Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 6 | 2A (Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 7 | 2B (Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| ... | ... |
| 25 | 5E (Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 26 - 254 | Reserved |
| 255 | Missing |
The table version control and its update needs to be managed by SC-MINT, so we would appreciate comments. I would be happy to have a short meeting to discuss this.
OkparaOkechukwu
commented
3 years ago Though I am working at the Nigerian Meteorological Agency (NIMET) PortHarcourt Airport Station, I would really like to work on a project like this for Nigeria, if given the opportunity.
efucile
commented
3 years ago Though I am working at the Nigerian Meteorological Agency (NIMET) PortHarcourt Airport Station, I would really like to work on a project like this for Nigeria, if given the opportunity.
@OkparaOkechukwu can I ask you to send an email to wis@wmo.int so that we can see if we can arrange for you to participate?
sergioh-pessoal
commented
3 years ago @DenisStuber, @jitsukoh, @richardweedon
I Agree with the 2 items
0-08-094 Method used to calculate the average daily temperature. -> Changing from 5 bits to 8 bits.
0-08-095 and 0-08-096 → Changing the form of representation From an element with 2 characters to a code table (8bits)However, concerning item 2, and taking into account Richard's comments in the meeting, I think that, in this case, it is also important that the code table can be understood without having to consult the manual.
In this case, the Term “1A-Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) does not help to understand what each classification means.
So, My suggestion would be something like: “1A - Flat, horizontal land, surrounded by an open space, / Uncertainty <0.2 K ((WMO-N8 2020 Edition)", etc.
jitsukoh
commented
3 years ago @sergioh-pessoal thank you for your feedback. I think @DavidBerryNOC 's comment was not exactly to avoid any reference to other resources... In this case, the definition depends on elements and is complex, so it might not be a good idea to extract only part of the definition... One idea would be to use the definition of the code registry but anyway we need to specify the exact definition for the sitting classification (e.g. CIMO Guide edition 2014 Annex 1B-1).
david-i-berry
commented
3 years ago @jitsukoh @sergioh-pessoal there were two issues in my comments.
I think the comments by @sergioh-pessoal address this but with a caveat. We need to be aware that the definitions are variable specific, this then leads to either a large code table with the classifications repeated but for the different variables or the addition of multiple code tables. Here the pragmatic choice would be to go with the suggestion by @jitsukoh but at the cost of (1), with the users not familiar with the classification needing to look up the meaning in WMO-No. 8 or other documents.
OkparaOkechukwu
commented
3 years ago @jitsukoh and @marijanacrepulja Here is a sample of the Chardonix station data encoded with the latest template and using the proposed descriptor 3-07-075. Note that this is only a preliminary sample. The quality control part needs more work. Chamonix_202012_307075.bufr.gz
Please which software did you use in encoding the data?
sergioh-pessoal
commented
3 years ago @OkparaOkechukwu, @jitsukoh, @marijanacrepulja
The library used to encode the message was the library of MBUFRTOOLS of INPE-BRAZIL. A simple FORTRAN program was written to read the Chardonix text file (provided by David) and encode it in the BUFR format using the MBUFRTOOLS library.
The BUFR tables of the program had to be modified. A new table ( version 36) was provisionally created and the proposed new descriptors was added inside this table to allow the software generate this sample.
Any BUFR software is expected to be able to decode this sample. However, as the descriptors do not officially exist yet, regardless of the software used, it will be necessary to manually create / modify the tables to add the new descriptors that are informed in the proposal, so that the tests can be carried out.
Modifications were made to the proposal after the creation of this BUFR sample. Thus, this sample is already out of date.
I take the opportunity to send an update of this sample (attached file). It has the same data provided before, however, the descriptors below have been modified as follows
008094 is now encoded with 8 bits 008095 and 008096 now are code tables with 8 bits.
The decoded version in text format is attached too.
Chamonix_202012_307075-v2.bufr.gz Chamonix_202012_307075-v2.decoded.txt.gz
OkparaOkechukwu
commented
3 years ago @sergioh-pessoal thanks for the clarification. Good practice: Whenever a BUFR file or message (or sample) is provided or mentioned (for the first time), it's good to also mention the software (if possible, including the software version) used in encoding/decoding it. This can go along way to help the BUFR "technical" community.
marianmajan-ibl
commented
3 years ago @jitsukoh and @sergioh-pessoal,
Where is the latest version of the sequence D descriptor 3-07-075 available, please? I imported the table from Jitsuko's comment from 19 April, however it differs from Sergio's decoded data. In Sergio's data there is the date sequence 3 01 011 right after the WIGOS and WMO ID sequence. In Jitsuko's table there is the Year/Month/Day sequence (incorrectly marked with the sequence number 3-01-001) after the 0-08-094 descriptor.
marianmajan-ibl
commented
3 years ago @jitsukoh and @sergioh-pessoal,
there was one additional discrepancy between the table above and the example – in the table the operator 2 04 004 was used while in the example the associated field width was 7 bit. With all the table D changes included, I was able to decode the sample bulletin. The values decoded by our software seems to be the same as in the dump provided by @sergioh-pessoal. The whole dump is attached: Chamonix_202012_307075-v2.decoded.ibl.txt I am also adding the 3 07 075 table in our internal format (it is CSV but needs to have the row id included) -- daycli_D.csv
jitsukoh
commented
3 years ago @DavidBerryNOC thank you for the clarification. It seems that the definitions are too complex to include all of them in the table (although it would be more convenient for users) so let us take the second option... @marianmajan-ibl my apologies, I have not had time to update the proposal above; here it is my understanding. Let me check with @sergioh-pessoal for discrepancies that you pointed out:
1. Add a new entry in the significance qualifier class of table B (0-08-094) and a new code table to describe the method used to calculate the daily average temperature.
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-094 | Method used to calculate the average daily temperature | Code table | 0 | 0 | 8 |
Discussion: See above.
CODE TABLE 0-08-094 0-08-094 Method used to calculate the average daily temperature
| Code Figure | |
|---|---|
| 0 | Average of maximum and minimum values: Tm = ( Tx + Tn) / 2 |
| 1 | Average of the 8 tri-hourly observation |
| 2 | Average of 24 hourly observation |
| 3 | Weighted average of 3 observations: Tm = (aT1 +bT2 + cT3), |
| 4 | Weighted average of 3 observation and also maximum and minimum values: Tm= (aT1 +bT2 + cT3 +dTx + eTn) |
| 5 | AWS complete integration from minute data |
| 6 - 254 | Reserved |
| 255 | Missing value |
Note : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x ”And“ n represent the values measured at different times or maximum (x) or minimum (n) values.
2. Quality flag for each climatological value (Temperatures, Precipitation and Snow). Add a new entry (5) of 4-bit indicator of quality control in the Associated field significance (0 31 021) (code table is tentative) (the code figure was changed from 10 to 5 because 5 seems a closer fit.)
| Code Figure | Code table | |
|---|---|---|
| 3 - 4 | reserved | |
| 5 | 4-bit indicator of quality control | 0 = Not controlled, 1 = Controlled and correct, 2 = Controlled and suspect, 3-14 = Reserved, 15 = Missing value |
Action item for SC-CLI/ET-DRC The code table will be finalized by SC-CLI/ET-DRC.
3. Siting classification and Measurement Quality Classification
Discussion: It is necessary to add the Siting Classification (SC) and also Measurement Quality Classification (MQC). However, the MQC must be accompanied by the siting classification.
Proposal:
Add 2 entries in table B: One for Temperature and another for Precipitation. Both with 8-bit code tables, where, the first character represents the Siting Classification from "1" to "5", as defined by defined ISO/WMO standard 119289:2014(E) (see the Guide to Instruments and Methods of Observation (WMO-No. 8) edition 2014 Part I, Chapter I, Annex 1B for details), and the second character represent the Measurement Quality Classification from “A” to “E” defined by the Guide to Instruments and Methods of Observation (WMO-No. 8) edition 2020.
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-095 | Siting and Measurement Quality Classification for temperature | Code table | 0 | 0 | 8 |
| 0-08-096 | Siting and Measurement Quality Classification for precipitation | Code table | 0 | 0 | 8 |
| CODE TABLE for 0-08-095 and 0-08-096 Code Figure | |
|---|---|
| 0 | Reserved |
| 1 | 1A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 2 | 1B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 3 | 1C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 4 | 1D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 5 | 1E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 6 | 2A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 7 | 2B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| ... | ... |
| 25 | 5E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 26 - 254 | Reserved |
| 255 | Missing |
Note: It is good to have these metadata in each message instead of having them only in OSCAR/Surface. Note: At this moment, there is not requirement for this kind of descriptor for snow.
4. Element names for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) Update the element name for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) to add terminology used in WIGOS Metadata.
| F X Y | Current | Updated (proposal) |
|---|---|---|
| 0 13 012 | Depth of fresh snow | Depth of fresh snow (Depth of snowfall) |
| 0 13 013 | Total snow depth | Total snow depth (Snow Depth) |
5. The new DAYCLI BUFR message
Update descriptor 3-07-075 (Update the sequence or create a new descriptor in the next version of BUFR table D)
| Table reference | Table references | Element name | Description / comments |
|---|---|---|---|
| 3-07-075 | Supplemental daily temperature and precipitation values for monthly climate report | ||
| 3-01-150 | WIGOS identifier | ||
| 3-01-001 | WMO block and station number | ||
| 3-01-021 | Latitude/longitude (high accuracy) | ||
| 0-07-030 | Height of station ground above mean sea level (see note 3) (m) | ||
| 0-08-095 | Siting and measurement quality classification for temperature | Code table - Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-096 | Siting and measurement quality classification for precipitation | Code table - Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-094 | Method used to calculate the average daily temperature | Code table | |
| 3-01-011 | Year/Month/Day | ||
| (Total accumulated precipitation) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-004 | Add associated field | 4 bits long | |
| 0-31-021 | Associated Field Significance | Set as 5 for 4-bit indicator of quality control | |
| 0-13-060 | Total accumulated precipitation (kg m-2) | ||
| 2-04-000 | Add associated field | cancel | |
| (Depth of fresh snow) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-004 | Add associated field | 4 bits long | |
| 0-31-021 | Associated Field Significance | Set as 5 for 4-bit indicator of quality control | |
| 0-13-012 | Depth of fresh snow (Depth of snowfall) | ||
| 2-04-000 | Add associated field | cancel | |
| (Total snow depth) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-004 | Add associated field | 4 bits long | |
| 0-31-021 | Associated Field Significance | Set as 5 for 4-bit indicator of quality control | |
| 0-13-013 | Total snow depth (Snow depth) | ||
| 2-04-000 | Add associated field | cancel | |
| (Max, Min, Mean temperature) | |||
| 0-07-032 | Height of sensor above local ground | for temperature measurement | |
| 1-07-003 | Replicate 7 descriptors 3 times | ||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 0-08-023 | First-order statistics | 2 – maximum; 3 – minimum; 4 – mean | |
| 2-04-004 | Add associated field | 4 bits long | |
| 0-31-021 | Associated Field Significance | Set as 5 for 4-bit indicator of quality control | |
| 0-12-101 | Temperature/air temperature (k) | ||
| 2-04-000 | Add associated field | cancel | |
| 0-08-023 | First-order statistics (code table) | Set as missing value |
6. Use of subset In this message, each subset is used to represent data for each day. Normally each subset is used to represent data from each station. Some guidance will be necessary to standardize how to encode messages from different stations.
jitsukoh
commented
3 years ago @sergioh-pessoal following up the ET-DRC meeting, I'd be grateful if you could provide a new sample data.
sergioh-pessoal
commented
3 years ago Dear @jitsukoh and @marianmajan-ibl
I checked the table posted on May 5th and it is correct. Then I checked the inconsistencies between the table and the sample posted on May 3rd. In fact, there were some inconsistencies. I'm sorry for this. I corrected the inconsistencies in the generation of the samples, and now I think it is correct. Attached is the sample (bufr and txt), based on the table posted on May 5th.
Chamonix_202012_v20210505.bufr.gz Chamonix_202012_v20210505.decoded.txt.gz
sergioh-pessoal
commented
3 years ago According to the meet of ET-DRC last week, I believe that item 2 of propose could be update like this.
2. Quality flag for each climatological value (Temperatures, Precipitation and Snow). Add a new entry (5) of 4-bit indicator of quality control in the Associated field significance (0 31 021) (code table is tentative)
| Code Figure | Code table | |
|---|---|---|
| 3 - 4 | reserved | |
| 5 | 4-bit indicator of quality control | 0 = Parameter is not measured at the station, 1 = Daily value not provided, 2 = Data unchecked, 3= Data checked and declared doubtful, 4= Data checked and declared aggregated, 5 =Data checked and declared out of instrument range, 6= Data checked and declared aggregated and out of instrument range, 7=Data checked and declared correct, 8-14 = Reserved, 15 = Missing value (QC info not available) |
Notes:
a) The Code figure 4 are used when observations are not available for some individual days, but the aggregated total (or highest/lowest value in the case of temperature) are known.
b) The Code figure 6 is similar to the figure 4, but the real values is higher the reported value.
c) In Both code figure 4 and 6, a sequence of days with missing values are provided and at the end of the sequence the total aggregaded value through during previous days.
Example: In the example bellow, because of an observer problem the rain gauge has accumulated a precipitation over 4 days (from DAY 2 to DAY 5 in the below tables). From DAY 2 to DAY 5 the QC information is set to 5 “Data checked and declared aggregated”. That allows specifying on how many days the aggregation corresponds to. For Day 6 there is no precipitation.
| Date | Precipitation | Associated QC information | QC code |
|---|---|---|---|
| Day 1 | 200 mm | Data checked and declared out of instrument range | 5 |
| Day 2 | Missing | Data checked and declared aggregated | 4 |
| Day 3 | Missing | Data checked and declared aggregated | 4 |
| Day 4 | Missing | Data checked and declared aggregated | 4 |
| Day 5 | 32.5 | Data checked and declared aggregated | 4 |
| Day 6 | 0 | Data checked and declared correct | 7 |
jitsukoh
commented
3 years ago @sergioh-pessoal below is the summary of yesterday's discussion:
@amilan17 we are still waiting for confirmation from ET-DRC but need to start working on the branch so can you create a branch for this (I moved this issue to in validation status)?
sergioh-pessoal
commented
3 years ago Hello @jitsukoh. Thank you very much for this information. If you prefer, I can update the proposal on git. I'll just wait for confirmation from @DenisStuber
jitsukoh
commented
3 years ago @sergioh-pessoal would you be kind enough to finalize the proposal 1) on the original comment of this issue and 2) input them on CSVs of the branch? I feel I'm asking you too much... in addition to the sample files etc... I think we need to decide which information can be included in MoC and which are not and need to go to a guidance document for DAYCLI producers to be developed by ET-DRC (e.g. notes to method of calculate average temperature and notes to quality information; notes cause headache for the Secretariat). I am sure the example of the use of "aggregated" quality information will go to the guidance document. I am happy to discuss a little on this.
jitsukoh
commented
3 years ago @sergioh-pessoal I have consolidated the proposal. Would you check it and let me know if you find issues? If it looks good, I will ask the Secretariat to help us update the branch.
1. Add a new entry in the significance qualifier class of table B (0-08-094) and a new code table to describe the method used to calculate the daily average temperature.
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-094 | Method used to calculate the average daily temperature | Code table | 0 | 0 | 8 |
CODE TABLE 0-08-094 0-08-094 Method used to calculate the average daily temperature
| Code Figure | -- |
|---|---|
| 0 | Average of maximum and minimum values: Tm = ( Tx + Tn) / 2 (see Note 1) |
| 1 | Average of the 8 tri-hourly observation |
| 2 | Average of 24 hourly observation |
| 3 | Weighted average of 3 observations: Tm = (aT1 +bT2 + cT3) (see Note 1) |
| 4 | Weighted average of 3 observation and also maximum and minimum values: Tm= (aT1 +bT2 + cT3 +dTx + eTn) (see Note 1) |
| 5 | AWS complete integration from minute data |
| 6 - 254 | Reserved |
| 255 | Missing value |
Note (1) : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x" and "n" represent the values measured at different times or maximum (x) or minimum (n) values.
2. Quality flag for each climatological value (Temperatures, Precipitation and Snow). Add a new entry (5) of 8-bit indicator of quality control in the Associated field significance (0 31 021).
| Code Figure | Code table | |
|---|---|---|
| 3 - 4 | reserved | |
| 5 | 8-bit indicator of quality control | 0 = Data checked and declared good, 1 = Data checked and declared suspect, 2 = Data checked and declared aggregated, 3 = Data checked and declared out of instrument range, 4 = Data checked and declared aggregated and out of instrument range, 5 = Parameter is not measured at the station, 6 = Daily value not provided, 7 = Data unchecked, 8-254 = Reserved, 255 = Missing (QC info not available) |
3. Siting classification and Measurement Quality Classification
Discussion: It is necessary to add the Siting Classification (SC) and also Measurement Quality Classification (MQC). However, the MQC must be accompanied by the siting classification.
Proposal:
Add 2 entries in table B: One for Temperature and another for Precipitation. Both with 8-bit code tables, where, the first character represents the Siting Classification from "1" to "5", as defined by defined ISO/WMO standard 119289:2014(E) (see the Guide to Instruments and Methods of Observation (WMO-No. 8) edition 2014 Part I, Chapter I, Annex 1B for details), and the second character represent the Measurement Quality Classification from “A” to “E” defined by the Guide to Instruments and Methods of Observation (WMO-No. 8) edition 2020.
| F X Y | ELEMENT NAME | UNIT | SCALE | REFERENCE VALUE | DATA WITH (bits) |
|---|---|---|---|---|---|
| 0-08-095 | Siting and Measurement Quality Classification for temperature | Code table | 0 | 0 | 8 |
| 0-08-096 | Siting and Measurement Quality Classification for precipitation | Code table | 0 | 0 | 8 |
| CODE TABLE for 0-08-095 and 0-08-096 Code Figure | |
|---|---|
| 0 | Reserved |
| 1 | 1A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 2 | 1B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 3 | 1C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 4 | 1D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 5 | 1E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 6 | 2A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 7 | 2B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| ... | ... |
| 25 | 5E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 26 - 254 | Reserved |
| 255 | Missing |
4. Element names for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) Update the element name for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) to add terminology used in WIGOS Metadata.
| F X Y | Current | Updated (proposal) |
|---|---|---|
| 0 13 012 | Depth of fresh snow | Depth of fresh snow (Depth of snowfall) |
| 0 13 013 | Total snow depth | Total snow depth (Snow Depth) |
5. The new DAYCLI BUFR message
Update descriptor 3-07-075 (Update the sequence or create a new descriptor in the next version of BUFR table D)
| Table reference | Table references | Element name | Description / comments |
|---|---|---|---|
| 3-07-075 | Supplemental daily temperature and precipitation values for monthly climate report | ||
| 3-01-150 | WIGOS identifier | ||
| 3-01-001 | WMO block and station number | ||
| 3-01-021 | Latitude/longitude (high accuracy) | ||
| 0-07-030 | Height of station ground above mean sea level (see note 3) (m) | ||
| 0-08-095 | Siting and measurement quality classification for temperature | Code table - Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-096 | Siting and measurement quality classification for precipitation | Code table - Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-094 | Method used to calculate the average daily temperature | Code table | |
| 3-01-011 | Year, Month, Day | ||
| (Total accumulated precipitation) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-008 | Add associated field | 8 bits long | |
| 0-31-021 | Associated field significance | Set as 5 for 8-bit indicator of quality control | |
| 0-13-060 | Total accumulated precipitation (kg m-2) | ||
| 2-04-000 | Add associated field | cancel | |
| (Depth of fresh snow) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-008 | Add associated field | 8 bits long | |
| 0-31-021 | Associated field significance | Set as 5 for 8-bit indicator of quality control | |
| 0-13-012 | Depth of fresh snow (Depth of snowfall) | ||
| 2-04-000 | Add associated field | cancel | |
| (Total snow depth) | |||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 2-04-008 | Add associated field | 8 bits long | |
| 0-31-021 | Associated field significance | Set as 5 for 8-bit indicator of quality control | |
| 0-13-013 | Total snow depth (Snow depth) | ||
| 2-04-000 | Add associated field | cancel | |
| (Max, Min, Mean temperature) | |||
| 0-07-032 | Height of sensor above local ground | for temperature measurement | |
| 1-07-003 | Replicate 7 descriptors 3 times | ||
| 0-04-004 | Hour | ||
| 0-04-024 | Time period or displacement | in hours | |
| 0-08-023 | First-order statistics | 2 – maximum; 3 – minimum; 4 – mean | |
| 2-04-008 | Add associated field | 8 bits long | |
| 0-31-021 | Associated field significance | Set as 5 for 8-bit indicator of quality control | |
| 0-12-101 | Temperature/air temperature (k) | ||
| 2-04-000 | Add associated field | cancel | |
| 0-08-023 | First-order statistics (code table) | Set as missing value |
sergioh-pessoal
commented
3 years ago Hi @jitsukoh I was trying to understand how to update the branch. But I didn't get it so far. Would be great if Secretariat help us to do this.
I have a observation concerning to Siting classification and Measurement Quality Classification. Denis has explained that there are possibilities that some places can have Siting Classification and not have Measurement Quality Classification or vice versa
As we discussed later, It is bather do not create more descriptors. So, in this case, code table for 0-08-095 and 0-08-096 need more 10 code figures to represent that possibilities. Another detail: It is bather start with code figure 0 and displacing the other elements on this table
Here is my suggestion for the update of the code table for 0-08-095 and 0-08-096
| Code Figure | |
|---|---|
| 0 | 1A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 1 | 1B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 2 | 1C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 3 | 1D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 4 | 1E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 5 | 2A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 6 | 2B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 7 | 2C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 8 | 2D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 9 | 2E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 10 | 3A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 11 | 3B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 12 | 3C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 13 | 3D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 14 | 3E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 15 | 4A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 16 | 4B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 17 | 4C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 18 | 4D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 19 | 4E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 20 | 5A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 21 | 5B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 22 | 5C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 23 | 5D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 24 | 5E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 26 | 1 (Siting Classification according to ISO/WMO standard 119289:2014(E) |
| 27 | 2 (Siting Classification according to ISO/WMO standard 119289:2014(E) |
| 28 | 3 (Siting Classification according to ISO/WMO standard 119289:2014(E) |
| 29 | 4 (Siting Classification according to ISO/WMO standard 119289:2014(E) |
| 30 | 5 (Siting Classification according to ISO/WMO standard 119289:2014(E) |
| 31 | A (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 32 | B (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 33 | C (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 34 | D (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 35 - 254 | Reserved |
| 255 | Missing |
jitsukoh
commented
3 years ago @sergioh-pessoal thank you for your proposal. Indeed it is necessary to have 10 more figures to cover possibilities of missing data. Regarding the allocation of code figures, I think using 1-5, 5-10, 11-15, ... cycle as code figures is intuitive and thus would help avoid errors... so my suggestion would be a table below. And yes, I will ask for help of the Secretariat to update the branch!
| Code Figure | |
|---|---|
| 0 | Reserved |
| 1 | 1A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 2 | 1B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 3 | 1C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 4 | 1D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 5 | 1E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 6 | 2A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 7 | 2B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 8 | 2C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 9 | 2D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 10 | 2E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 11 | 3A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 12 | 3B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 13 | 3C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 14 | 3D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 15 | 3E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 16 | 4A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 17 | 4B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 18 | 4C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 19 | 4D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 20 | 4E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 21 | 5A (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 22 | 5B (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 23 | 5C (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 24 | 5D (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 25 | 5E (Siting Classification according to ISO/WMO standard 119289:2014(E) and Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition) |
| 26 | 1 (Siting Classification according to ISO/WMO standard 119289:2014(E), Measurement Quality Classification is missing |
| 27 | 2 (Siting Classification according to ISO/WMO standard 119289:2014(E), Measurement Quality Classification is missing |
| 28 | 3 (Siting Classification according to ISO/WMO standard 119289:2014(E), Measurement Quality Classification is missing |
| 29 | 4 (Siting Classification according to ISO/WMO standard 119289:2014(E), Measurement Quality Classification is missing |
| 30 | 5 (Siting Classification according to ISO/WMO standard 119289:2014(E), Measurement Quality Classification is missing |
| 31 | A (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition), Siting Classification is missing |
| 32 | B (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition), Siting Classification is missing |
| 33 | C (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition), Siting Classification is missing |
| 34 | D (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition), Siting Classification is missing |
| 35 | E (Measurement Quality Classification according to the Guide to Instruments and Methods of Observation (WMO-No. 8), 2020 Edition), Siting Classification is missing |
| 36 - 254 | Reserved |
| 255 | Missing |
jitsukoh
commented
3 years ago @amilan17 @chenxiaoxia2019 I posted the finalized proposal in the original post. Could you help us update the branch according to the proposal?
chenxiaoxia2019
commented
3 years ago @jitsukoh @amilan17 Well noted. I will update the branch.
chenxiaoxia2019
commented
3 years ago @jitsukoh @amilan17 The branch has been updated according to the finalized proposal.
jitsukoh
commented
3 years ago @chenxiaoxia2019 thank you very much for updating the branch. @sergioh-pessoal when you prepare sample data, can you ask Richard and Marian to decode the sample data using the tables in the branch?
sergioh-pessoal
commented
3 years ago @jitsukoh Here there are the BUFR samples and the corresponding dumped text file @marianmajan-ibl and @richardweedon could you decoded these sample ? Thank you
daycli_meteofrance_stations-test.bufr.decoded.txt.gz daycli_meteofrance_stations-test.bufr.gz daycli-test-us-72315.bufr.decoded.txt.gz daycli-test-us-72315.bufr.gz daycli-test-Curitiba-Brazil.bufr.decoded.txt.gz daycli-test-Curitiba-Brazil.bufr.gz
jitsukoh
commented
3 years ago @sergioh-pessoal thank you so much for working on this... @marianmajan-ibl and @richardweedon the branch is already updated; you can use the tables for decoding. @DenisStuber I would appreciate it if you distribute the sample to your community to see if the data includes what is needed.
sergioh-pessoal
commented
3 years ago @jitsukoh, @marianmajan-ibl , @richardweedon , @DenisStuber
I'm updating the samples because some errors were detected in the previous codifications Please consider the samples bellow
daycli_meteofrance_stations-test.bufr.decoded.txt.gz daycli_meteofrance_stations-test.bufr.gz daycli-test-Curitiba-Brazil.bufr.decoded.txt.gz daycli-test-Curitiba-Brazil.bufr.gz daycli-test-us-72315.bufr.decoded.txt.gz daycli-test-us-72315.bufr.gz
DenisStuber
commented
3 years ago I just realise that some countries as India or Luxembourg are measuring some parameters not at full hour. e.g. at 06:15 or at 11:45. Sergio would it be possible to modify the format of the TX-HOUR, TN_HOUR, RR_HOUR, DS_HOUR, and TSD_HOUR to allow the precision until the minute ?
jitsukoh
commented
3 years ago I remember we discussed the issue and ruled out the possibility, i.e. keep the hour precision, when we discussed the time displacement. Of course we can accommodate it technically but it further increases the complexity and I don't think we can make it to the next fast-track cycle with the change. But it is up to the community.
richardweedon
commented
3 years ago I may be premature in light of the request from Dennis, but please see attached decodes of the test data as requested. I am in the process of discussing the new sequence and the decode of the test data with our Climate Data teams. I will provide any feedback as it becomes available. daycli_meteofrance_stations-test-output.txt daycli-test-us-72315-output.txt daycli-test-Curitiba-Brazil-output.txt
jitsukoh
commented
3 years ago @richardweedon thank you very much for the decoded data. ET-DRC has found other possibilities of time slots and we decided to take more time to validate this sequence with a wider variety of samples. At this moment we are not sure if we need to update the sequence or not, but we might need to ask the team for another round of validation.
sergioh-pessoal
commented
3 years ago @richardweedon Thank you for the decoded data.
jitsukoh
commented
3 years ago @sergioh-pessoal @DenisStuber here is a new proposed sequence, considering the existing practices. The representation of the time period for each parameter is updated so that it always specifies the beginning time (allowing to represent the previous day using time displacement in days), assuming that the period is always 24 hours.
The new DAYCLI BUFR message
Update descriptor 3-07-075 (Update the sequence or create a new descriptor in the next version of BUFR table D)
| Table reference | Table references | Element name | Description / comments |
|---|---|---|---|
| 3-07-075 | Supplemental daily temperature and precipitation values for monthly climate report | ||
| 3-01-150 | WIGOS identifier | ||
| 3-01-001 | WMO block and station number | ||
| 3-01-021 | Latitude/longitude (high accuracy) | ||
| 0-07-030 | Height of station ground above mean sea level (see note 3) (m) | ||
| 0-08-095 | Siting and measurement quality classification for temperature | Code table - Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-096 | Siting and measurement quality classification for precipitation | Code table - Examples: 1A,1B, 2A,2B,... etc. | |
| 0-08-094 | Method used to calculate the average daily temperature | Code table | |
| 3-01-011 | Year, Month, Day | ||
| (Total accumulated precipitation) | |||
| 0-04-023 | Time period or displacement | in day since reference date, 0 (when beginning time of the period is on the same day) or -1 (previous day) | |
| 3-01-013 | Hour, minute, second | beginning time of the period | |
| 2-04-008 | Add associated field | 8 bits long | |
| 0-31-021 | Associated field significance | Set as 5 for 8-bit indicator of quality control | |
| 0-13-060 | Total accumulated precipitation (kg m-2) | ||
| 2-04-000 | Add associated field | cancel | |
| (Depth of fresh snow) | |||
| 0-04-023 | Time period or displacement | in day since reference date, 0 (when beginning time of the period is on the same day) or -1 (previous day) | |
| 3-01-013 | Hour, minute, second | beginning time of the period | |
| 2-04-008 | Add associated field | 8 bits long | |
| 0-31-021 | Associated field significance | Set as 5 for 8-bit indicator of quality control | |
| 0-13-012 | Depth of fresh snow (Depth of snowfall) | ||
| 2-04-000 | Add associated field | cancel | |
| (Total snow depth) | |||
| 0-04-023 | Time period or displacement | in day since reference date, 0 (when beginning time of the period is on the same day) or -1 (previous day) | |
| 3-01-013 | Hour, minute, second | beginning time of the period | |
| 2-04-008 | Add associated field | 8 bits long | |
| 0-31-021 | Associated field significance | Set as 5 for 8-bit indicator of quality control | |
| 0-13-013 | Total snow depth (Snow depth) | ||
| 2-04-000 | Add associated field | cancel | |
| (Max, Min, Mean temperature) | |||
| 0-07-032 | Height of sensor above local ground | for temperature measurement | |
| 1-07-003 | Replicate 7 descriptors 3 times | ||
| 0-04-023 | Time period or displacement | in day since reference date, 0 (when beginning time of the period is on the same day) or -1 (previous day) | |
| 3-01-013 | Hour, minute, second | beginning time of the period | |
| 0-08-023 | First-order statistics | 2 – maximum; 3 – minimum; 4 – mean | |
| 2-04-008 | Add associated field | 8 bits long | |
| 0-31-021 | Associated field significance | Set as 5 for 8-bit indicator of quality control | |
| 0-12-101 | Temperature/air temperature (k) | ||
| 2-04-000 | Add associated field | cancel | |
| 0-08-023 | First-order statistics (code table) | Set as missing value |
sergioh-pessoal
commented
3 years ago Hi Denis and Jitsuko. Here is the sample encoded in BUFR and its decoding in text format. 20210906_Example_Toulouse_Blagnac_07630_version_2.bufr.gz 20210906_Example_Toulouse_Blagnac_07630_version_2.bufr.txt.gz
DenisStuber
commented
3 years ago Thanks a lot Sergio. I will ask the persons in charge at Météo-France for validation ...
----- Météo-France ----- STUBER DENIS DCSC/EDR @.*** Fixe : +33 561078337
De: "Sergkl - Sergio Ferreira" @.> À: "wmo-im" @.> Cc: "Denis Stuber" @.>, "Mention" @.> Envoyé: Vendredi 10 Septembre 2021 13:56:13 Objet: Re: [wmo-im/BUFR4] BUFR sequence for DAYCLI (#51)
Hi Denis and Jitsuko. Here is the sample encoded in BUFR and its decoding in text format. [ https://github.com/wmo-im/BUFR4/files/7143554/20210906_Example_Toulouse_Blagnac_07630_version_2.bufr.gz | 20210906_Example_Toulouse_Blagnac_07630_version_2.bufr.gz ] [ https://github.com/wmo-im/BUFR4/files/7143555/20210906_Example_Toulouse_Blagnac_07630_version_2.bufr.txt.gz | 20210906_Example_Toulouse_Blagnac_07630_version_2.bufr.txt.gz ]
— You are receiving this because you were mentioned. Reply to this email directly, [ https://github.com/wmo-im/BUFR4/issues/51#issuecomment-916846943 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/ALLN7B325VOZ26NSD2YDJGTUBHW53ANCNFSM4R7FMFXA | unsubscribe ] . Triage notifications on the go with GitHub Mobile for [ https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 | iOS ] or [ https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub | Android ] .
david-i-berry
commented
3 years ago Decoded file attached, decoding using custom python BUFR decoder. Link to code below.
I've only checked the first subset and this appears to match that in the text file. I haven't checked the others.
Python script/app using flask
https://github.com/DavidBerryNOC/daycli_validation
(comment edited to add link to repo+code used for decoding)
jitsukoh
commented
3 years ago @amilan17 @chenxiaoxia2019 I updated the original post of this issue (far above) to reflect the final proposal of the sequence 3-07-075 (exactly the same as the one I posted on August 6). Only change I made is the sequence 3-07-075. Could you help update the branch (rebase the branch if necessary and update the branch)?
Summary and purpose
Background, history and reasoning for the reporting of daily climate observations The development of the principal measure of the state of the climate - the global temperature record - has extensively depended on monthly CLIMAT data provided by National Meteorological and Hydrological Services (NMHSs). Over the last 20 years, there has been a growing demand for indices and measures of the climate that also consider extremes (Jones et al., 2012). For many extreme measures, monthly data are insufficient and there is a need for operationally exchanged daily climate data. This need is not just for timeliness, but principally for data that is compatible with long historical daily series developed and made available by NMHSs. Attempts have been made to use SYNOP data for this purpose (e.g. by the European Climate Assessment and Dataset but there are serious issues of incompatibility of SYNOP data with traditional methods of climate measurement within NMHSs (see van den Besselaar et al., 2012). Daily summaries in SYNOP messages are based on measurements that occur between synoptic reporting times and often over a period of less than 24 hours. For instance, in Europe, minimum temperatures are recorded usually over the 18 to 06 UTC 12-hour period and maximum temperatures during the 06 to 18 UTC 12-hour period. Measured in this way, the true daily minimum and maximum temperatures may not be reported because they may have occurred outside those particular 12-hour periods. As a result, SYNOP reports have been shown to significantly underestimate extremes: minimum temperatures measured in this way may be higher than the true daily minimum temperature, and maximum temperatures reported may be lower than the true daily maximum temperature reported as 24-hour climate observation. Similar problems occur for precipitation. In other regions of the world, SYNOP reporting practices can differ but problems remain. The Commission for Basic Systems (CBS) Open Programme Area Group on Integrated Observing Systems (OPAG-IOS), Implementation/Coordination Team on Integrated Observing Systems (ICT-IOS), recommended in 2012 that daily climate observations be included in monthly CLIMAT reports as a means of addressing the gap in the quality of daily climate observations. The U.S. National Oceanic and Atmospheric Administration (NOAA) National Centers for Environmental Information (NCEI), in cooperation with WMO Inter-programme Expert Team on Data Representation Maintenance and Monitoring (IPET-DRMM) and NOAA National Centers for Environmental Prediction (NCEP), developed a BUFR template for transmission of daily climate observations in BUFR format. This template was approved by CBS for implementation in May 2015. It was subsequently tested in the United States, with the cooperation of the UK Met Office. A one-year trial phase for the monthly reporting of daily climate observations was accepted by delegates to the seventeenth session of the Commission for Climatology in April 2018 (see Recommendation 5 (CCl-17)).
Reporting daily climate observations: Technical solution NOAA/NCEI, in cooperation with IPET-DRMM (taken over by the Inter-programme Expert Team on Codes Maintenance (IPET-CM) in 2016) and NOAA/NCEP, developed a BUFR template, 3 07 074 - Supplemental daily temperature and precipitation values, for daily climate observations in BUFR format, for monthly reporting. Please note that this does not replace the existing CLIMAT BUFR templates but offers complementary reporting of daily observations once per month. BUFR template 3 07 074 enables NMHSs to provide 31 daily observations consistent with national climate databases for the following elements: • Time of observation for temperature • Daily maximum temperature • Daily minimum temperature • Daily mean temperature (if it differs from (Tmax+Tmin)/2) • Time of observation for precipitation • Total daily precipitation • Depth of new snowfall • Depth of total snow on the ground Each of these observations should be recorded at the observing time consistent with the climate reporting practices of the NMHS and should reflect conditions over the previous 24-hour period. The climate convention varies from country to country; each country should retain its traditional observing practice in reporting daily climate summaries. For example, while in the U.S. the reporting time is local midnight, in Australia it is 9 a.m. local, and in Canada it is 06 UTC. These observations can be efficiently provided via daily CLIMAT reports or other methods specifically designed for climate purposes.
Consultations and Reviewers
References Jones, P.D., Lister, D.H., Osborn, T.J., Harpham, C., Salmon, M., Morice, C.P., 2012: Hemispheric and large-scale land-surface air temperature variations: An extensive revision and an update to 2010. Journal of Geophysical Research, 117, D05127, doi:10.1029/2011JD017139.
Van den Besselaar, E.J.M., Klein Tank, A.M.G, van der Schrier, G. and Jones, P.D., 2012: Synoptic messages to extend climate data records. Journal of Geophysical Research, 117, D07101, doi:10.1029/2011JD1688.
Detailed proposal
(Final proposal, updated by @jitsukoh December 20.) 1. Add a new entry in the significance qualifier class of table B (0-08-094) and a new code table to describe the method used to calculate the daily average temperature.
CODE TABLE 0-08-094 0-08-094 Method used to calculate the average daily temperature
Note (1) : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x" and "n" represent the values measured at different times or maximum (x) or minimum (n) values.
2. Quality flag for each climatological value (Temperatures, Precipitation and Snow). Add a new entry (5) of 8-bit indicator of quality control in the Associated field significance (0 31 021).
3. Siting classification and Measurement Quality Classification
Discussion: It is necessary to add the Siting Classification (SC) and also Measurement Quality Classification (MQC). However, the MQC must be accompanied by the siting classification.
Proposal:
Add 2 entries in table B: One for Temperature and another for Precipitation. Both with 8-bit code tables, where, the first character represents the Siting Classification from "1" to "5", as defined by defined ISO/WMO standard 119289:2014(E) (see the Guide to Instruments and Methods of Observation (WMO-No. 8) edition 2014 Part I, Chapter I, Annex 1B for details), and the second character represent the Measurement Quality Classification from “A” to “D” defined by the Guide to Instruments and Methods of Observation (WMO-No. 8) edition 2020.
4. Element names for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) This change will not be implemented. ~~Update the element name for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) to add terminology used in WIGOS Metadata.~~
* CSV file: BUFRCREX_TableB_en_13.csv0 13 012Depth of fresh snowDepth of fresh snow (Depth of snowfall)0 13 013Total snow depthTotal snow depth (Snow depth) (see Note 2)5. The new DAYCLI BUFR message Create a new descriptor 3-07-075 in the next version of BUFR table D.
(Depth of snowfall)(Snow depth)