Revision of E-Profile Sequences

[richardweedon]

Branch

https://github.com/wmo-im/BUFR4/tree/issue-21a

Summary and purpose

A windprofiler (WP) is a ground based remote sensing instrument to measure vertical profiles of wind, this can be either a radar, lidar or sodar. Radar based WP networks exist worldwide and the data are encoded in BUFR to be exchanged over the GTS. However, each of these networks uses a different implementation of the BUFR code. In this document we propose a harmonized implementation of BUFR for WP data. The proposal is based on a review of the existing implementations and on data user requirements. The purpose of this template is for the exchange of Level 2 product data for Meteorological Products and not for the distribution of L0 or L1 products.

As part of the COST action ES0702 EG-CLIMET http://www.eg-climet.org , a special working group was formed to look at the “Harmonisation of European automatic (elastic backscatter) Lidar and Ceilometer Network, Calibrations, recording formats and archiving protocols”. As part of these activities it was agreed that a common BUFR template should be agreed upon for this data. After an extensive review the following Table D sequence has been agreed upon by the E-PROFILE ALC Expert Team.

These proposals are an initiative of the E-PROFILE programme of EUMETNET. This was discussed at the E-PROFILE ET in Helsinki 15-17th October.

Action proposed

final proposal from Sep 2020: https://github.com/wmo-im/BUFR4/files/5306953/wind_prof_prp-20200930.docx

Full details of the new (revised 5th june 2016) D sequences and Descriptors are given in the attached document windproflidar-amend-2_bis+MT.docx

[SibylleK]

@efucile What is the status of this proposal? Do you think it is still possible to get it validated for the fast-track? The EUMETNET E-PROFILE team has a strong requirement for the new sequences. The document is worth to be discussed and agreed to by the team. My comments are, that I do not think that new descriptors are needed to minimize the data width by 1 bit (all "Uncertainty in..") 3 01 032 for "Common header sequence" has to be changed, because it already exists. But, it seems to be a typo and should be 3 01 132. Apart from that, I would agree to the new proposed sequences and descriptors. If there is no strong objection to the proposal, could we get a branch for the validation? And if so, @richardweedon could you provide some example for the validation here?

[efucile]

@SibylleK and @richardweedon I think we should try to validate this. There is strong pressure to do it and it is pending for many years. It is time to close it. However, @jitsukoh has to decide if we can push it in this fast track. I think that we are going to be late for other reasons and this could give us time to finalize this.

[jitsukoh]

@efucile, @SibylleK and @richardweedon, thank you for the proposal and my apologies to have not replied earlier. It took some time to refresh my memory on this... I reviewed our discussion in 2016 and honestly I am a little worried about repeating the same thing. The core issue of the discussion back in 2016 was the representation of uncertainty; on one hand the team was looking for a generic way of expressing it (not only for elements of profiler but for radiosonde) to avoid using too many descriptors and on the other hand, the detailed definition and the way of calculation of proposed uncertainty descriptor for E-EPROFILE (i.e. how to use the proposed descriptors) was not given. I posed same questions over and over again because JMA's profiler experts did not understand how the information would be provided and how they would be used in a meaningful way, but I didn't get any information and I ended up accepting them because blocking the process was not my intention. My question was about the descriptors that are removed in this new proposal. So in short, we wasted 3 descriptors. I think our experts would have the same questions over the uncertainty descriptors (and maybe other questions) and I would like to make sure this point is clarified this time before making a decision. Can I suggest that we set a deadline, for example by the end of next week for posting technical questions on GitHub and ask the proposer to answer them also on GitHub, and if all the technical issues are solved, we have a short teleconference to conclude? At this moment, I don't think we have time on 22nd to discuss details on this proposal. I'd appreciate your consideration.

[simonebircher]

@jitsukoh , thank you very much for your efforts spent on the E-PROFILE BUFR template proposal as well as you comments posted here. We appreciate your suggestion to answer technical questions on GitHub by the end of next week and a concluding teleconference.

Could you possibly provide us with a link to the discussion in 2016 that you are referring to? Thanks a lot in advance and best regards

Simone Bircher-Adrot E-PROFILE assistant network manager

[efucile]

Dear @simonebircher we welcome your participation in this forum. I understand that you are working for Eumetnet and you are involved in the E-Profile network, but I don't have your details in the WMO expert database. We can take contributions only from experts registered in the WMO or we accept opinions from invited experts. I would kindly ask you to send an email to me efucile@wmo.int and we can try to formalize your intervention.

[jitsukoh]

Dear team members, @wmo-im/tdcf

We got an urgent proposal from EUMETNET, which was not on the agenda on our teleconference in May, and I would like to propose following arrangements to accommodate the request. Because of the nature of this proposal which requires an intensive review, I would like to invite all of you to review the proposal and post questions to this forum by June 17, but please let me know if you cannot make it (need more time to review), because this arrangement is an exception of the schedule that the team had already agreed. I ask @richardweedon and other colleagues to answer posted questions, and when all the technical issues are solved on Github, I will propose a teleconference to conclude. Thank you for your support in advance.

[SibylleK]

@jitsukoh Dear Jisukoh, I have already made some comments regarding the new descriptors for the uncertainties. I think, they are not needed as they are only minimize the data width by 1 bit of the existing one. I suppose they are only proposed to have the descriptors in a pretty order. The other new descriptors are for changing the reference values to allow negative values. Although I am not at all in favor of changing existing entries, it is also possible to change the reference values for the existing entries in BUFR table version 35. This means of course that the correct version number had to be used by encoding and decoding such BUFR. For one entry (Particle depolarization ratio) the bit width would change, which makes the BUFR unreadable if a wrong BUFR table version is used. There is a typo at the descriptor for"Common header sequence". It should be 3 01 132. Within the Common header sequence 0 02 003 -"Type of measuring equipment used" is used instead of the new proposed 0 02 005. It this intended? The new sequences do not include the Common header sequence. Normally it would be included in such sequences. But of course they could be preceded by the header sequence.

If the requirement is such urgent and the messages should be on GTS as soon as possible, they could be encoded with a pure sequence of descriptors, of course. Despite of the new entries for the "Upper air remote sensing instrument type" it could be encoded already by using operator descriptors. But, I know, changing the reference value with the usage of 203YYY causes trouble in some decoding software....

These are my more technical comments to the proposal and I am now going on holiday until the 6th of July, therefore I am sorry, that I am not able to respond or support the validation process till then.

[marijanacrepulja]

I agree with @SibylleK comments on the new descriptors for the uncertainties, as I believe we can use existing descriptors without introducing new elements for reducing data width. When comes to new descriptors for introducing new reference value, we can use operator 203YYY and existing descriptors for this purpose. However, if there is an issue that 203 operator can cause trouble in some decoding software we can address this with new descriptors.

@richardweedon I have a question regarding 0 33 002. Does quality information (0 33 002) refer to u, v, w component respectively in the new sequence 3 09 024? Similar in the sequence 3 09 025, quality information (0 33 002) refers to virtual temperature and w component respectively? When comes to sequence 3 09 026, does quality information (0 33 002) refer to entire message or?

[jitsukoh]

This is the question that I posed in 2016, but could you provide us with any reference document(s) about the calculation of measurement uncertainties that you are using in producing the wind profiler and lidar products?

[jitsukoh]

Dear team members, @wmo-im/tdcf

To accommodate the request for approval of E-Profile sequences by the next fast-track procedure, a separate teleconference will be held on July 8 at 12UTC. Enrico already sent an invitation to those who already posted comments, but if there is anyone interested in this discussion and decision making, s/he is welcome to join, so please let me know.

[jbathegit]

@jitsukoh , @efucile , @SibylleK , @marijanacrepulja , @richardweedon

I'm interested in this topic; however, I'm not available on July 8th at 12 UTC because I have a separate telecon (for GODEX-NWP) from 11-14 UTC on that same day. Could I just ask you to please keep me in the loop on any developments or final decisions, because we ingest wind profiler data at NCEP, so we'll want to begin preparing our software for the new wind profiler sequence at whatever time it is planned to start being used on the GTS. (Thanks!)

[jitsukoh]

@jbathegit sorry for having scheduled the meeting without notice. It was an urgent request and we needed to adjust our procedure. I will make sure all the questions are answered on Github and also discussions at the teleconference will be recorded here to keep you in the loop.

[richardweedon]

The Revised versions of the proposal along with a summary of the questions asked in the intial review of the proposal with the corresponding answers is attached

[richardweedon]

BUFR responses.docx windproflidar-amend-20200624.docx

[richardweedon]

see attached

[sergioh-pessoal]

I am interested in this topic and would like to attend the meeting on the 8th at 12 UTC. Thanks

[jitsukoh]

@jbathegit A summary of the discussion today is here. The sequence will be amended and presented to the team as part of validation process.

[jitsukoh]

@chenxiaoxia2019 attached is the revised version of the BUFR sequence provided by @richardweedon based on our discussion on Wednesday. Could you create a branch for this proposal and pose questions if there are anything unclear? wind_prof_prp_MT_AH.docx

[chenxiaoxia2019]

@jitsukoh Thanks. I have already created the branch for this issue. I will make changes according to the new version which is more clear. I will get you updated about the progress. @richardweedon

[SibylleK]

In the updated proposal the existing descriptors were changed in reference value and data width. But I thought it was "preferred to create new elements". What should be the finale decision here? In addition the definition of the new descriptor 0 02 006 - "Upper Air Remote Sensing Instrument Type" is missing.

[marijanacrepulja]

I believe we decided to create new elements with reference to windproflidar-amend-20200624.docx We also wanted to add 0 08 021 Time significance and set it to 2 with regards to date and time repeated twice in the header sequence.

[jitsukoh]

Thank you @SibylleK and @marijanacrepulja , yes, you're right. We decided to introduce new descriptors instead of changing existing ones and also use 0 08 021 time significance in the common header sequence. @richardweedon , could you update the document before producing sample data?

[richardweedon]

Marijana Apologies for the misunderstanding in the original proposal we will re-issue with the new Table B descriptors as originally discussed. We were less clear on the inclusion of the Significance qualifier , if we change the test data at this moment in time it will require more work than originally envisioned.

The document put out by Enrico makes no mention of this. Seehttps://github.com/wmo-im/CCT/wiki/Teleconference-8.7.2020

Regards

Richard

From: Marijana Crepulja notifications@github.com Sent: 15 July 2020 17:25 To: wmo-im/BUFR4 BUFR4@noreply.github.com Cc: Weedon, Richard richard.weedon@metoffice.gov.uk; Mention mention@noreply.github.com Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21)

I believe we decided to create new elements with reference to windproflidar-amend-20200624.docx We also wanted to add 0 08 021 Time significance and set it to 2 with regards to date and time repeated twice in the header sequence.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/wmo-im/BUFR4/issues/21#issuecomment-658863046, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5TCSP3SWNZPPQB4OURI73R3XJ4PANCNFSM4MZMWKYQ.

[richardweedon]

Jitsukoh , Sibylle , Marijana I will amend the original proposal as below –

Changes to Table B Descriptors

BUFR

CREX

TABLE REFERENCE

DATA

DATA

ELEMENT NAME UNIT SCALE REFERENCE WIDTH UNIT SCALE WIDTH F* X Y

VALUE (Bits)

Characters 0 15 073 replaces 0 15 063 Attenuated Backscatter m –1 Sr -1

8 -524288 20 m –1 Sr -1 8 7 0 15 074 replaces 0 15 065 Particle Backscatter Coefficient m –1 Sr -1 8 -524288 20 m –1 Sr -1 8 7 0 15 075 replaces 0 15 067 Particle Extinction Coefficient m1 8 -524288

20 m1 8 7 0 15 076 replaces 0 15 069 Particle LIDAR Ratio Sr 2 -2048 13 Sr 1 5 0 15 077 replaces 0 15 070 Uncertainty in LIDAR Ratio Sr 2 0 12 Sr 1 5 0 15 078 replaces 0 15 071 Particle Depolarization Ratio % 2 -8192 15 % 2 5

The changes above will require new D sequences as below. The proposal will be changed accordingly

309024 to replace 309021 309026 to replace 309023

I am in the process of amending the original proposal will re-issue this afternoon.

Regards

Richard

From: jitsukoh notifications@github.com Sent: 16 July 2020 06:46 To: wmo-im/BUFR4 BUFR4@noreply.github.com Cc: Weedon, Richard richard.weedon@metoffice.gov.uk; Mention mention@noreply.github.com Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21)

Thank you @SibylleKhttps://github.com/SibylleK and @marijanacrepuljahttps://github.com/marijanacrepulja , yes, you're right. We decided to introduce new descriptors instead of changing existing ones and also use 0 08 021 time significance in the common header sequence. @richardweedonhttps://github.com/richardweedon , could you update the document before producing sample data?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/wmo-im/BUFR4/issues/21#issuecomment-659173021, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5TCSITPHKZXUE5CTEFCTTR32HYFANCNFSM4MZMWKYQ.

[SibylleK]

As I understand it the new proposed Table B descriptors would be:

FXY	ElementName_en	Note_en	BUFR_Unit	BUFR_Scale	BUFR_ReferenceValue	BUFR_DataWidth_Bits	CREX_Unit	CREX_Scale	CREX_DataWidth_Char
002006	Upper air remote sensing instrument type	Code table	0	0	6	Code table	0	0	2
015073	Attenuated Backscatter	m-1 sr-1	8	-524288	20	m-1 sr-1	8	7
015074	Particle Backscatter Coefficient	m-1 sr-1	8	-524288	20	m-1 sr-1	8	7
015075	Particle Extinction Coefficient	m	8	-524288	20	m	8	7
015076	Particle LIDAR Ratio	sr	2	-2048	13	sr	1	5
015077	Uncertainty in LIDAR Ratio	sr	2	0	12	sr	1	5
015078	Particle Depolarization Ratio	%	2	-8192	15	%	2	5

@richardweedon could you confirm this?

By the way, I would also propose to keep the existing descriptor 015070 - "Uncertainty in lidar ratio" with 14 bits, so that 015077 would not be needed.

It would ease the validation, if the new descriptors were added in the branch.

[richardweedon]

Sibylle Have just sent the attached to Enrico, this should answer your questions. Many thanks for your help.

Regards

Richard

From: Sibylle Krebber notifications@github.com Sent: 16 July 2020 16:20 To: wmo-im/BUFR4 BUFR4@noreply.github.com Cc: Weedon, Richard richard.weedon@metoffice.gov.uk; Mention mention@noreply.github.com Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21)

As I understand it the new proposed Table B descriptors would be: FXY ElementName_en Note_en BUFR_Unit BUFR_Scale BUFR_ReferenceValue BUFR_DataWidth_Bits CREX_Unit CREX_Scale CREX_DataWidth_Char 002006 Upper air remote sensing instrument type Code table 0 0 6 Code table 0 0 015073 Attenuated Backscatter m-1 sr-1 8 -524288 20 m-1 sr-1 8 7 015074 Particle Backscatter Coefficient m-1 sr-1 8 -524288 20 m-1 sr-1 8 7 015075 Particle Extinction Coefficient m 8 -524288 20 m 8 7 015076 Particle LIDAR Ratio sr 2 -2048 13 sr 1 5 015077 Uncertainty in LIDAR Ratio sr 2 0 12 sr 1 5 015078 Particle Depolarization Ratio % 2 -8192 15 % 2 5

@richardweedonhttps://github.com/richardweedon could you confirm this?

By the way, I would also propose to keep the existing descriptor 015070 - "Uncertainty in lidar ratio" with 14 bits, so that 015077 would not be needed.

It would ease the validation, if the new descriptors were added in the branch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/wmo-im/BUFR4/issues/21#issuecomment-659482216, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5TCSI73AUOETGTANMKDSLR34LCFANCNFSM4MZMWKYQ.

[richardweedon]

In relation to the comments below we have agreed the suggested changes, please see the attached for an updated version of the tables –

From: Sibylle Krebber notifications@github.com Sent: 16 July 2020 16:20 To: wmo-im/BUFR4 BUFR4@noreply.github.com Cc: Weedon, Richard richard.weedon@metoffice.gov.uk; Mention mention@noreply.github.com Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21)

As I understand it the new proposed Table B descriptors would be: FXY ElementName_en Note_en BUFR_Unit BUFR_Scale BUFR_ReferenceValue BUFR_DataWidth_Bits CREX_Unit CREX_Scale CREX_DataWidth_Char 002006 Upper air remote sensing instrument type Code table 0 0 6 Code table 0 0 015073 Attenuated Backscatter m-1 sr-1 8 -524288 20 m-1 sr-1 8 7 015074 Particle Backscatter Coefficient m-1 sr-1 8 -524288 20 m-1 sr-1 8 7 015075 Particle Extinction Coefficient m 8 -524288 20 m 8 7 015076 Particle LIDAR Ratio sr 2 -2048 13 sr 1 5 015077 Uncertainty in LIDAR Ratio sr 2 0 12 sr 1 5 015078 Particle Depolarization Ratio % 2 -8192 15 % 2 5

@richardweedonhttps://github.com/richardweedon could you confirm this?

By the way, I would also propose to keep the existing descriptor 015070 - "Uncertainty in lidar ratio" with 14 bits, so that 015077 would not be needed.

It would ease the validation, if the new descriptors were added in the branch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/wmo-im/BUFR4/issues/21#issuecomment-659482216, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5TCSI73AUOETGTANMKDSLR34LCFANCNFSM4MZMWKYQ.

[chenxiaoxia2019]

@richardweedon Hi, Richard, new branch for this issue has been created. And all the changes have been made according to the proposal you sent. Could you please check it? Thanks. wind_prof_prp-2[3272].docx

[richardweedon]

Hi I have been asked to make a last minute change the details are summarised in the attached

Thanks

Richard

From: xchen notifications@github.com Sent: 17 July 2020 12:51 To: wmo-im/BUFR4 BUFR4@noreply.github.com Cc: Weedon, Richard richard.weedon@metoffice.gov.uk; Mention mention@noreply.github.com Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21)

@richardweedonhttps://github.com/richardweedon Hi, Richard, new branch for this issue has been created. And all the changes have been made according to the proposal you sent. Could you please check it? Thanks. wind_prof_prp-2[3272].docxhttps://github.com/wmo-im/BUFR4/files/4937553/wind_prof_prp-2.3272.docx

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/wmo-im/BUFR4/issues/21#issuecomment-660065118, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5TCSP3KAFZHC52VXB74WTR4A3INANCNFSM4MZMWKYQ.

[marijanacrepulja]

I have spotted in github that units for 015075 Particle Extinction Coefficient is m. Previously defined 015067 Particle extinction coefficient has units m-1. Should we have m-1 for 015075?

[richardweedon]

My apologies we have corrected the units to 015067 (changed to m-1) in the attached. This is an editorial change and will not affect the test data. May I ask that this be included on the Branch

Regards

Richard

From: xchen notifications@github.com Sent: 17 July 2020 12:51 To: wmo-im/BUFR4 BUFR4@noreply.github.com Cc: Weedon, Richard richard.weedon@metoffice.gov.uk; Mention mention@noreply.github.com Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21)

@richardweedonhttps://github.com/richardweedon Hi, Richard, new branch for this issue has been created. And all the changes have been made according to the proposal you sent. Could you please check it? Thanks. wind_prof_prp-2[3272].docxhttps://github.com/wmo-im/BUFR4/files/4937553/wind_prof_prp-2.3272.docx

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/wmo-im/BUFR4/issues/21#issuecomment-660065118, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5TCSP3KAFZHC52VXB74WTR4A3INANCNFSM4MZMWKYQ.

[jitsukoh]

@richardweedon the descriptor that should be corrected is 015075. I am not updated about the validation status, but I don't believe the use of time significance descriptor included in wind_prof_prp-2[3272].docx is correct, and it is not included in the branch. It is difficult to use the data if the meaning of time is not clear, so I would suggest that you correct this. Because we cannot wait more than a few days, we need to give up including this in FT2020-2 if it takes more than that to agree on the sequence and complete the validation.

I got the comment below from @SibylleK about the use of the time significance:

Regarding the usage of the time significance for indicate "values are averaged over time", from my point of view it should rather precede the measured or calculated values than the date and time. If a time significance for the time is required the entry 28 - "Start of scan" and 29 - "End of scan or..." would suit, but this could not be used within 301014.

But BUFR regulation 94.5.3.4 ("The consecutive occurrence of two identical element descriptors or identical sets of element descriptors from Classes 04 to 07 inclusive shall denote a range of values bounded by the corresponding element values. This enables the definition of layers and simple time periods.") already defines, that with the sequence 301014, i.e. the replication of date and time, a time period is indicated, although it is not used very often. More usual is the time significance (2=Time averaged) in combination with a "Time period of displacement". (e.g. see IUPC.. RJTD messages)

The time significant should go in front of 011003 - "u-component" in sequence 3 09 024 and 012007 -"Virtual temperature" in sequence 309025. I do not know if the "attenuated backscatter..." and are also mean values.. A proper "place" for the cancellation has to be assigned, too. This depends, if it is correct to handle "quality information" etc. also as "averaged values" or not.

[sergioh-pessoal]

I also thing that is not clear the meaning of Time Significance in the common sequence 3-01-132.

The Time significance is often followed by descriptors such as date and time or time period or displacement.

In my opinion, one possibility would be to remove the replicator 1-02-002 from the common sequence (3-01-132), and include the “Time Significance” , and “Time Period or Displacement” in the appropriated place in each templates 3-09-024, 3-09-025, 3- 09-026.

Sergio

[jitsukoh]

I also think the approach suggested by @SibylleK and @sergioh-pessoal is natural. I made the first proposal here but I am not quite sure if these additional descriptors (0 08 021 and 0 04 025 and cancellation) should be repeated, as I don't know which elements are time averaged. If all the elements inside the replication is time averaged, they should go around the replication. I would appreciate comments.

(Just for reference, here is the format of Japan's wind profile data cited by Sibylle, using 0 08 021 (time significance set to 2; time averaged) and 0 04 025 (time period or displacement set to -10 mins))

@richardweedon I tried to aggregate all the changes made so far in the Word document but please confirm that everything is right, as the post did not include a file.

[chenxiaoxia2019]

My apologies we have corrected the units to 015067 (changed to m-1) in the attached. This is an editorial change and will not affect the test data. May I ask that this be included on the Branch

Regards

Richard

From: xchen notifications@github.com Sent: 17 July 2020 12:51 To: wmo-im/BUFR4 BUFR4@noreply.github.com Cc: Weedon, Richard richard.weedon@metoffice.gov.uk; Mention mention@noreply.github.com Subject: Re: [wmo-im/BUFR4] Revision of E-Profile Sequences (#21)

@richardweedonhttps://github.com/richardweedon Hi, Richard, new branch for this issue has been created. And all the changes have been made according to the proposal you sent. Could you please check it? Thanks. wind_prof_prp-2[3272].docxhttps://github.com/wmo-im/BUFR4/files/4937553/wind_prof_prp-2.3272.docx

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#21 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5TCSP3KAFZHC52VXB74WTR4A3INANCNFSM4MZMWKYQ.

@richardweedon The units of 015075 has been updated in the branch, changing from m to m-1.

[jitsukoh]

@chenxiaoxia2019 could you update the branch according to wind_prof_prp-20200722.docx, but only the new table B descriptors (the scale of 015076 and 015077 was changed from 2 to 1 in validation process, and the width of 015077 should be corrected to 12) and remove all the changes in table D entries from this branch? We separate the approval of table D descriptors (301132, 309024, 309025, 309026) from FT2020-2 and will approve them in the FT2021-1 cycle.

=== The following is for record. === E-PROFILE project team and our team agreed on the solution for the time significance: 1) Remove the replication of time from the common header sequence 301132 2) Use 008021 (=29 "End of scan") to specify the end of time period in the common header sequence 301132 3) Use 008021 (=2 "Time averaged") with 004025 (negative value, being used with the end of time period) in each data section (309024, 309025, 309026), before the replication because it is generally not height dependent.

And it was agreed to move 008092 and 008093 in 309026 to the beginning of the back scatter data section.

In the process of validation, the issue of bit width of mean frequency was reported by DWD:

When I try to validate this BUFR I got to another problem. I am not able to decode the data with one of our operational BUFR software package, as it does not allow integer values greater than 32 bits. But the bit widths of "Mean frequency" in the proposal is set to 37. (and scale to -5) Unfortunately, if such an accuracy is really needed and included in the sequences, we won't be able to use it internally in the foreseeable future.

To resolve the issue, the teams agreed to the approach:

1. We approve all the new Table B descriptors in FT2020-2 process. They will be in operational status in November.
2. We discuss and resolve the issue of the bit width of mean frequency by September 18.
3. We approve the new Table D descriptors (301132, 309024, 309025, 309026) in FT2021-1 process. They will be in operational status in May 2021.

[chenxiaoxia2019]

@jitsukoh The branch for this issue has been updated according to the latest proposal, with only new entries in Table B Descriptors. @richardweedon Could you please check this branch? Thanks.

[SibylleK]

For the validation of the new table B descriptors, here are sample data provided by Myles Turp last week.

ALC_VALIDATION.zip RWP_VALIDATION.zip

[SibylleK]

I used the tables of the branch for this issue for the decoding of the BUFR. As the mean frequency in the sample BUFR is encoded with 37 bits, I can only provide the output of our Java-decoding software.

ALC_DWD_OUTPUT.zip RWP_DWD_OUTPUT.zip

Disregarding some difficulties and differences in the representation of the data values, the decoded values are equal.

[SibylleK]

But I changed the reference value of 0 15 077 to 0 as it is seems to be still -2048 in the branch.

[chenxiaoxia2019]

@SibylleK Hi, Sibyllek. Thanks for your reminder. The reference of 015077 has been fixed.

[sergioh-pessoal]

I decoded the sample from Myles sent by Sibilly using my software. No problems in the decoding

See output attached I’ve noted that there is a typo in the common header Sequence in the document wind_prof_prp-2.3272.

The descriptor 3-01-150 contains only ( 0-01-125, 0-01-126, 0-01-127. 0-01-128) The position of these descriptor in the 3 collum of document is correct, but the descriptors from 3-01-011 to 0-07-039 are apparently shifted to the right from where they should be ALC_CPTEC_OUTPUT.zip RWP_CPTEC_OUTPUT.zip

[jitsukoh]

E-Profile team provided an updated proposal as wind_prof_prp-20200731.docx, using two different SCALE/WIDTH combinations to avoid a bid width more than 32 for "mean frequency."

@marijanacrepulja has expressed her support on this solution and I'd appreciate others' opinions, especially @SibylleK .

[jitsukoh]

Based on the comment of @marijanacrepulja E-Profile team updated the proposal so that the mean frequency in 3 09 026 sequence is outside of the replication (same as in 3 09 024 and 3 09 025 sequences). I'd appreciate confirmation of @marijanacrepulja and @SibylleK !

From the E-Profile team: Please see attached revised version: wind_prof_prp-20200918.docx with changes highlighted in green. Please confirm that the mean frequency descriptors are now in the appropriate location in sequence 3 09 026 and I will generate the updated test messages and send to the validators.

[marijanacrepulja]

@jitsukoh It seems to be correct now.

[SibylleK]

@jitsukoh From my point of view, it is OK too.

I have only one comment regarding the note "...must be used with the Common header sequence 3010132 and cannot be used on its own" under the sequences 309024, 309025 and 309026. Is it intended to include this as element description or note in table D? If so, I would rather propose to include 3010132 directly in the sequences. Or the note should be changed to a recommendation, so that the sequences could be used in combination with other "header sequences".

Are there any comments, preferences or already decisions on this issue? Because, for the validation of the new sequences, to be done for FT2021-I, an updated branch would be appreciated.

[jitsukoh]

@SibylleK thank you for your comments. Regarding the note around header sequence 301132, here are comments from E-Profile team:

it would still be useful to keep this separate as there are plans to including this for future E-PROFILE data templates rather than integrate into longer sequences.

I'd suggest updating the note to something like "The common header sequence 3 01 132 cannot be used on its own and must be followed by one of the sequences for windprofiler product data."

Regarding Table D descriptors - please find attached an example for ALC sequence using just the Table D sequences and set as bufr table version 36. I am having some issues with producing the same data for RWP sequence, however if you feel this will be useful I will investigate further and produce some more sample data.

@chenxiaoxia2019 could you create a new branch for the new Table D descriptors, because these are on a separate amendment process.

[SibylleK]

Dear @jitsukoh and all, Meanwhile the E-Profile team provides a revised version of the E-Profile products sequences. wind_prof_prp-20200930.docx

The common header sequence 301132 is now included in the sequences 309024, 309025, 309026. I upload it here, because some sample data for the Single wavelength Elastic Backscatter Lidar sequence (309026) already provided for the validation.

Please find attached: ALC_VALIDATION_TABLED.zip

[SibylleK]

Attached the output of the DWD decoding software could be found. The values are the same as in the provided .txt files.

ALC_VALIDATION_TABLED_DWDout.zip

The validation was done manually (including the new sequence by typewriting). It would be good to repeat it using the new branch for this issue.

[jitsukoh]

Dear @SibylleK thank you very much for the update (and my apologies for the misunderstanding of your comment on the note!).

@wmo-im/tdcf please let us know, by the end of this week, if you have opposition to the proposal on table D descriptors.

@chenxiaoxia2019 could you create a new branch that includes only the new table D descriptors?