new section 4 for encoding quantiles of post-processed products

[sebvi]

Initial request

This proposal suggests a combination of existing templates to encode post-processed quantile products. One of the applications for these templates is a post-processing method called ecPoint, which include quantile forecasts representing the distribution of sub-grid scale point values. Template 86, ‘Quantile forecasts at a horizontal level or in a horizontal layer at a point in time’ and template 87, ‘Quantile forecasts at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval’ are used to encode quantile forecasts. We propose two new templates based on 86 and 87 but with additional keys, which are already existing in the templates 70, 71, 72, 73, 93, 94, 97 and 98. These keys are the ‘Input process identifier, the ‘Input originating centre and the ‘Type of post-processing’.

Amendment details

ADD to Comon Code Table 4.0

Code	Description
89	Post-processed quantile forecasts at a horizontal level or in a horizontal layer at a point in time
90	Post-processed quantile forecasts at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

Add new template 4.89 "Post-processed quantile forecasts at a horizontal level or in a horizontal layer at a point in time"

Octet	Description
10	Parameter Category (see code table 4.1)
11	Parameter Number (see code table 4.2)
12-13	Input process identifier (see Note 1)
14-15	Input originating centre (see code table common/c-11 and Note 2)
16	Type of post-processing (see Note 3)
17	Type of generating process (see code table 4.3)
18	Background generating process identifier (defined by originating centre)
19	Forecast generating process identifier (defined by originating centre)
20-21	Hours after reference time of data cut-off
22	Minutes after reference time of data cut-off
23	Indicator of unit of time range (see code table 4.4)
24-27	Forecast time in units defined by octet 23
28	Type of first fixed surface (see code table 4.5)
29	Scale factor of first fixed surface
30-33	Scaled value of first fixed surface
34	Type of second fixed surface (see code table 4.5)
35	Scale factor of second fixed surface
36-39	Scaled value of second fixed surface
40-41	Total number of Quantiles
42-43	Quantile Value

Add new template 4.90 "Post-processed quantile forecasts at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval"

Octet	Description
10	Parameter Category (see code table 4.1)
11	Parameter Number (see code table 4.2)
12-13	Input process identifier (see Note 1)
14-15	Input originating centre (see code table common/c-11 and Note 2)
16	Type of post-processing (see Note 3)
17	Type of generating process (see code table 4.3)
18	Background generating process identifier (defined by originating centre)
19	Forecast generating process identifier (defined by originating centre)
20-21	Hours after reference time of data cut-off
22	Minutes after reference time of data cut-off
23	Indicator of unit of time range (see code table 4.4)
24-27	Forecast time in units defined by octet 23
28	Type of first fixed surface (see code table 4.5)
29	Scale factor of first fixed surface
30-33	Scaled value of first fixed surface
34	Type of second fixed surface (see code table 4.5)
35	Scale factor of second fixed surface
36-39	Scaled value of second fixed surface
40-41	Total number of Quantiles
42-43	Quantile Value
44-45	Year of end of overall time interval
46	Month of end of overall time interval
47	Day of end of overall time interval
48	Hour of end of overall time interval
49	Minute of end of overall time interval
50	Second of end of overall time interval
51	n - number of time range specifications describing the time intervals used to calculate the statistically processed field
52-55	Total number of data values missing in statistical process
	56-67 Specification of the outermost (or only) time range over which statistical  processing is done
56	Statistical process used to calculate the processed field from the field at each time  increment during the time range (see code table 4.10)
57	Type of time increment between successive fields used in the statistical processing (see code table 4.11)
58	Indicator of unit of time for time range over which statistical processing is done (see code table 4.4)
59-62	Length of the time range over which statistical processing is done, in units defined  by the previous octet
63	Indicator of unit of time for the increment between the successive fields used (see code table 4.4)
64-67	Time increment between successive fields, in units defined by the previous octet
	68-nn These octets are included only if n > 1, where nn = 55 + 12 x n
68-79	As octets 56 to 67, next innermost step of processing
80-nn	Additional time range specifications, included in accordance with the value of n.  Contents as octets 56 to 67, repeated as necessary

Notes common to both templates: 1) The input process identifier shall have the value of the ""analysis or forecast process identifier"" of the original GRIB message used as input of the post-processing. 2) The input originating centre shall have the value of the ""originating centre"" of the original GRIB message used as input of the post-processing. 3) This identifies which post-processing technique was used. This is defined by the originating centre.

Comments

No response

Requestor(s)

Robert Osinski, ECMWF Sebastien Villaume, ECMWF

Stakeholder(s)

ECMWF

Publication(s)

Manual on Codes (WMO-No. 306), Volume I.2, GRIB code table 4.0 Manual on Codes (WMO-No. 306), Volume I.2, GRIB section 4 template 4.89 Manual on Codes (WMO-No. 306), Volume I.2, GRIB section 4 template 4.90

Expected impact of change

MEDIUM

Collaborators

Timothy Hewson, ECMWF Fatima Pillosu, ECMWF

References

No response

Validation

an ecCodes branch, which implements this proposal, can be downloaded at: ecCodes branch for ecPoint data

[sebvi]

branch created, I am doing the modifs now

[sebvi]

branch updated

[amilan17]

@sebvi do you want this to go through FT2023-2?

[sebvi]

unless you tell me I can't, I would like to have it in

[marijanacrepulja]

@sebvi and @amilan17 I believe we can include it in FT2023-2. Lets discuss it with the team tomorrow.

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.13.July.2023 notes:

Sebastien will look at editorial comments in PR; note consistency can be addressed as needed in a different task

[amilan17]

@sebvi @marijanacrepulja @SibylleK -- I had a couple minor changes. you can view the commit here: https://github.com/wmo-im/GRIB2/commit/253dcfaa602f8a392dae01a85ef754161b80004d

[amilan17]

@SibylleK will you have a chance to validate this too?

[SibylleK]

@amilan17, @sebvi, There are no examples provided. Therefore I can't do a proper validation this week.

BUT, I checked the entries in the branch and made a little change: https://github.com/wmo-im/GRIB2/commit/ca247a3524452d8f660438a71721370d7443ab77

From my point of view the templates are Ok. I have a minor editorial remark: Regarding the entries of the Input process identifier, input originating centre and type of post-processing, I think it would be good to have these entries consitent with the other post-processed templates, e.g. 4.70, including the notes, which are missing in the proposal and the branch.

I also compared our PDT configurations produced from the branch with the provided eccodes template definitions. This comparison was OK.

Is this sufficient for a validation? Otherwise, could we complete the validation as an addendum?

[amilan17]

@sebvi do you have a chance to address Sibylle's comments above?

[sebvi]

I could try to have a look later today. In the worst case scenario I am back at work on Monday

[sebvi]

I have added the notes similar to template 4.70 as suggested by @SibylleK as it is a very good point.

If samples are really required I could produce some but not before Monday.

[amilan17]

Otherwise, could we complete the validation as an addendum?

@SibylleK @sebvi Thank you for your recent feedback. I will still bring this into FT2023-2, but please do validate asap. thx.

[amilan17]

@sebvi I had to do some cleanup with the notes. Can you please check them in FT2023-2? 

https://github.com/wmo-im/GRIB2/tree/FT2023-2

[sebvi]

If it is adding "see note 5" in the template, to be consistent with similar templates, then I am fine with it and agree it should be added