code table 4.0 and templates 4.92 - 4.98: Extend local time template to ensembles, postprocessed forecasts and average, accumulated or other statistically processed values

[sebvi]

Branch

https://github.com/wmo-im/GRIB2/tree/issue84

Summary and purpose

ECMWF is proposing a new set of templates to extend existing templates for use with local time

Proposer

Sebastien Villaume (ECMWF)

Action proposed

The team is kindly asked to review and approve the contents for inclusion within the next update to the WMO Manual on Codes.

Discussions

Following the recent validation of the new local time template (4.88), issue https://github.com/wmo-im/GRIB2/issues/6 , we would like to extend the concept to be used in EPS and in postprocessing models. WE also extend the local time to be be used with statistically processed parameters.

LocalTime.pptx

Detailed proposal

add in code Table 4.0 the entries for the new templates	Code	Meaning
92	Individual ensemble forecast, control and perturbed at a horizontal level or in a horizontal layer at a local Time
93	Post-processing analysis or forecast at a horizontal level or in a horizontal layer at a local Time
94	Post-processing individual ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a local Time
95	Average, accumulation, extreme values or other statistically processed values at a horizontal level or in a horizontal layer at a local Time
96	Average, accumulation, extreme values or other statistically processed values of an individual ensemble forecast, control and perturbed at a horizontal level or in a horizontal layer at a local Time
97	Average, accumulation, extreme values or other statistically processed values of a post-processing analysis or forecast at a horizontal level or in a horizontal layer at a local Time
98	Average, accumulation, extreme values or other statistically processed values of a post-processing individual ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a local Time

Template 4.92 Individual ensemble forecast, control and perturbed at a horizontal level or in a horizontal layer at a local Time

Octet No.	Contents
10	Parameter category (see Code table 4.1)
11	Parameter number (see Code table 4.2)
12	Type of generating process (see Code table 4.3)
13	Background generating process identifier (defined by originating centre)
14	Analysis or forecast generating process identifier (defined by originating centre)
15	Type of first fixed surface (see Code table 4.5)
16	Scale factor of first fixed surface
17-20	Scaled value of first fixed surface
21	Type of second fixed surface (see Code table 4.5)
22	Scale factor of second fixed surface
23-26	Scaled value of second fixed surface
27	Type of ensemble forecast (see Code table 4.6)
28	Perturbation number
29	Number of forecasts in ensemble
30	Method used to derive the data field values at the local time specified in section 1 (see code Table 2.248)
31	n - number of analysis or forecast used to create the composite data field at the local time specified in section 1 (n>=1)
	Octet 32-49 Specification of the analysis or forecast used in the processing (n=1)
32-33	Year of the analysis or forecast used in the processing
34	Month of the analysis or forecast used in the processing
35	Day of the analysis or forecast used in the processing
36	Hour of the analysis or forecast used in the processing
37	Minute of the analysis or forecast used in the processing
38	Second of the analysis or forecast used in the processing
39	Indicator of units of forecast time (see code Table 4.4, set to missing if analysis)
40-43	Forecast time (set to missing if analysis)
44	Number of time increments of the forecast used in the processing
45	Indicator of units of time for the time increments
46-49	Time increments between successive forecast time
	Octet 50-nn are included only if n>1 where nn=31+18*n
50-nn	(n-1) repetitions of sequence of octet 32-49, describing the next analyses or forecasts used in the processing

Template 4.93 Post-processing analysis or forecast at a horizontal level or in a horizontal layer at a local Time

Octet No.	Contents
10	Parameter category (see Code table 4.1)
11	Parameter number (see Code table 4.2)
12-13	Input process identifier
14-15	Input originating centre
16	Type of post-processing
17	Type of generating process (see Code table 4.3)
18	Background generating process identifier (defined by originating centre)
19	Analysis or forecast generating process identifier (defined by originating centre)
20	Type of first fixed surface (see Code table 4.5)
21	Scale factor of first fixed surface
22-25	Scaled value of first fixed surface
26	Type of second fixed surface (see Code table 4.5)
27	Scale factor of second fixed surface
28-31	Scaled value of second fixed surface
32	Method used to derive the data field values at the local time specified in section 1 (see code Table 2.248)
33	n - number of analysis or forecast used to create the composite data field at the local time specified in section 1 (n>=1)
	Octet 34-51 Specification of the analysis or forecast used in the processing (n=1)
34-35	Year of the analysis or forecast used in the processing
36	Month of the analysis or forecast used in the processing
37	Day of the analysis or forecast used in the processing
38	Hour of the analysis or forecast used in the processing
39	Minute of the analysis or forecast used in the processing
40	Second of the analysis or forecast used in the processing
41	Indicator of units of forecast time (see code Table 4.4, set to missing if analysis)
42-45	Forecast time (set to missing if analysis)
46	Number of time increments of the forecast used in the processing
47	Indicator of units of time for the time increments
48-51	Time increments between successive forecast time
	Octet 52-nn are included only if n>1 where nn=33+18*n
52-nn	(n-1) repetitions of sequence of octet 34-51, describing the next analyses or forecasts used in the processing

Template 4.94 Post-processing individual ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a local Time

Octet No.	Contents
10	Parameter category (see Code table 4.1)
11	Parameter number (see Code table 4.2)
12-13	Input process identifier
14-15	Input originating centre
16	Type of post-processing
17	Type of generating process (see Code table 4.3)
18	Background generating process identifier (defined by originating centre)
19	Analysis or forecast generating process identifier (defined by originating centre)
20	Type of first fixed surface (see Code table 4.5)
21	Scale factor of first fixed surface
22-25	Scaled value of first fixed surface
26	Type of second fixed surface (see Code table 4.5)
27	Scale factor of second fixed surface
28-31	Scaled value of second fixed surface
32	Type of ensemble forecast (see Code table 4.6)
33	Perturbation number
34	Number of forecasts in ensemble
35	Method used to derive the data field values at the local time specified in section 1 (see code Table 2.248)
36	n - number of analysis or forecast used to create the composite data field at the local time specified in section 1 (n>=1)
	Octet 37-54 Specification of the analysis or forecast used in the processing (n=1)
37-38	Year of the analysis or forecast used in the processing
39	Month of the analysis or forecast used in the processing
40	Day of the analysis or forecast used in the processing
41	Hour of the analysis or forecast used in the processing
42	Minute of the analysis or forecast used in the processing
43	Second of the analysis or forecast used in the processing
44	Indicator of units of forecast time (see code Table 4.4, set to missing if analysis)
45-48	Forecast time (set to missing if analysis)
49	Number of time increments of the forecast used in the processing
50	Indicator of units of time for the time increments
51-54	Time increments between successive forecast time
	Octet 55-nn are included only if n>1 where nn=36+18*n
55-nn	(n-1) repetitions of sequence of octet 37-54, describing the next analyses or forecasts used in the processing

TEMPLATE 4.95 Average, accumulation, extreme values or other statistically processed values at a horizontal level or in a horizontal layer at a local Time

Octet No.	Contents
10	Parameter category (see Code table 4.1)
11	Parameter number (see Code table 4.2)
12	Type of generating process (see Code table 4.3)
13	Background generating process identifier (defined by originating centre)
14	Forecast generating process identifier (defined by originating centre)
15	Type of first fixed surface (see Code table 4.5)
16	Scale factor of first fixed surface
17–20	Scaled value of first fixed surface
21	Type of second fixed surface (see Code table 4.5)
22	Scale factor of second fixed surface
23–26	Scaled value of second fixed surface
27	Statistical process used to calculate the fields that will be used in the local time processing (see Code table 4.10)
28	Indicator of unit of time for time range over which statistical processing is done. (see Code table 4.4)
29-32	Length of the time range over which statistical processing is done, in units defined by the previous octet. (see note 1)
33	Number of statistically processed fields used in the local time composite field (see Note 2)
34	Method used to calculate the field value at the local time specified in section 1 (see Code table 4.248)
35	n - number of forecasts used in the local time processing (see Note 3)
n/a	36-53 Specification of the first (or only) forecast
36-37	Year of the reference date for the first (or only) forecast
38	Month of the reference date for the first (or only) forecast
39	Day of the reference date for the first (or only) forecast
40	Hour of the reference time for the first (or only) forecast
41	Minute of the reference time for the first (or only) forecast
42	Second of the reference time for the first (or only) forecast
43	Indicator of units of forecast time (see Code table 4.4)
44-47	Forecast time in units defined by the previous octet
48	Number of increments for the successive forecast times
49	Indicator of unit of time for the time increment between the successive forecast times (see Code table 4.4)
50-53	Time increment between successive forecast times, in units defined by the previous octet (see Note 4)
n/a	54-nn These octets are included only if n>1 where nn=35+18*n
54-nn	(n-1) repetitions of sequence of octet 36-53, describing the next forecasts (reference date, reference time and forecast time increments)

Notes: (1) This represents the length of time over which the statistical processing was applied. The local time defined in section 1 represents the end of this processing. For instance, a value of 24h corresponds to a statistical processing between the previous day at local time and this day at local time. (2) This represents the number of statistically processed fields (or stripes) used to create the composite local time field. For instance a value of 8 means that 8 statistically processed fields have been used in the processing, each of them representing a section of 45 degrees of longitude (360/8) centered around the UTC time corresponding to the local time. (3) This is the number of forecasts and time steps used to create the statistically processed fields. These implicitly have the same statistical process as defined in octet 27. If a forecast has 2 time increments (3 hourly day 1 to 5 then 6 hourly), it should be encoded as 2 forecasts with the same reference time, using the appropriate starting forecast time and time increments. (4) This also represents the length of time range of the statistical processed fields. For instance, to create a 24h accumulation (encoded in octet 29-32), we could use several 3h accumulations, or 6h accumulation, a mixture of the two, etc.

TEMPLATE 4.96 Average, accumulation, extreme values or other statistically processed values of an individual ensemble forecast, control and perturbed at a horizontal level or in a horizontal layer at a local Time

Octet No.	Contents
10	Parameter category (see Code table 4.1)
11	Parameter number (see Code table 4.2)
12	Type of generating process (see Code table 4.3)
13	Background generating process identifier (defined by originating centre)
14	Forecast generating process identifier (defined by originating centre)
15	Type of first fixed surface (see Code table 4.5)
16	Scale factor of first fixed surface
17–20	Scaled value of first fixed surface
21	Type of second fixed surface (see Code table 4.5)
22	Scale factor of second fixed surface
23–26	Scaled value of second fixed surface
27	Type of ensemble forecast (see Code table 4.6)
28	Perturbation number
29	Number of forecasts in ensemble
30	Statistical process used to calculate the fields that will be used in the local time processing (see Code table 4.10)
31	Indicator of unit of time for time range over which statistical processing is done. (see Code table 4.4)
32-35	Length of the time range over which statistical processing is done, in units defined by the previous octet. (see note 1)
36	Number of statistically processed fields used in the local time composite field (see Note 2)
37	Method used to calculate the field value at the local time specified in section 1 (see Code table 4.248)
38	n - number of forecasts used in the local time processing (see Note 3)
n/a	39-56 Specification of the first (or only) forecast
39-40	Year of the reference date for the first (or only) forecast
41	Month of the reference date for the first (or only) forecast
42	Day of the reference date for the first (or only) forecast
43	Hour of the reference time for the first (or only) forecast
44	Minute of the reference time for the first (or only) forecast
45	Second of the reference time for the first (or only) forecast
46	Indicator of units of forecast time (see Code table 4.4)
47-50	Forecast time in units defined by the previous octet
51	Number of increments for the successive forecast times
52	Indicator of unit of time for the time increment between the successive forecast times (see Code table 4.4)
53-56	Time increment between successive forecast times, in units defined by the previous octet (see Note 4)
n/a	57-nn These octets are included only if n>1 where nn=38+18*n
57-nn	(n-1) repetitions of sequence of octet 39-56, describing the next forecasts (reference date, reference time and forecast time increments)

Notes: (1) This represents the length of time over which the statistical processing was applied. The local time defined in section 1 represents the end of this processing. For instance, a value of 24h corresponds to a statistical processing between the previous day at local time and this day at local time. (2) This represents the number of statistically processed fields (or stripes) used to create the composite local time field. For instance a value of 8 means that 8 statistically processed fields have been used in the processing, each of them representing a section of 45 degrees of longitude (360/8) centered around the UTC time corresponding to the local time. (3) This is the number of forecasts and time steps used to create the statistically processed fields. These implicitly have the same statistical process as defined in octet 30. If a forecast has 2 time increments (3 hourly day 1 to 5 then 6 hourly), it should be encoded as 2 forecasts with the same reference time, using the appropriate starting forecast time and time increments. (4) This also represents the length of time range of the statistical processed fields. For instance, to create a 24h accumulation (encoded in octet 32-35), we could use several 3h accumulations, or 6h accumulation, a mixture of the two, etc.

TEMPLATE 4.97 Average, accumulation, extreme values or other statistically processed values of a post-processing analysis or forecast at a horizontal level or in a horizontal layer at a local Time

Octet No.	Contents
10	Parameter category (see Code table 4.1)
11	Parameter number (see Code table 4.2)
12-13	Input process identifier
14-15	Input originating centre
16	Type of post-processing
17	Type of generating process (see Code table 4.3)
18	Background generating process identifier (defined by originating centre)
19	Forecast generating process identifier (defined by originating centre)
20	Type of first fixed surface (see Code table 4.5)
21	Scale factor of first fixed surface
22–25	Scaled value of first fixed surface
26	Type of second fixed surface (see Code table 4.5)
27	Scale factor of second fixed surface
28–31	Scaled value of second fixed surface
32	Statistical process used to calculate the fields that will be used in the local time processing (see Code table 4.10)
33	Indicator of unit of time for time range over which statistical processing is done. (see Code table 4.4)
34-37	Length of the time range over which statistical processing is done, in units defined by the previous octet. (see note 1)
38	Number of statistically processed fields used in the local time composite field (see Note 2)
39	Method used to calculate the field value at the local time specified in section 1 (see Code table 4.248)
40	n - number of forecasts used in the local time processing (see Note 3)
n/a	41-58 Specification of the first (or only) forecast
41-42	Year of the reference date for the first (or only) forecast
43	Month of the reference date for the first (or only) forecast
44	Day of the reference date for the first (or only) forecast
45	Hour of the reference time for the first (or only) forecast
46	Minute of the reference time for the first (or only) forecast
47	Second of the reference time for the first (or only) forecast
48	Indicator of units of forecast time (see Code table 4.4)
49-52	Forecast time in units defined by the previous octet
53	Number of increments for the successive forecast times
54	Indicator of unit of time for the time increment between the successive forecast times (see Code table 4.4)
55-58	Time increment between successive forecast times, in units defined by the previous octet (see Note 4)
n/a	59-nn These octets are included only if n>1 where nn=40+18*n
59-nn	(n-1) repetitions of sequence of octet 41-58, describing the next forecasts (reference date, reference time and forecast time increments)

Notes: (1) This represents the length of time over which the statistical processing was applied. The local time defined in section 1 represents the end of this processing. For instance, a value of 24h corresponds to a statistical processing between the previous day at local time and this day at local time. (2) This represents the number of statistically processed fields (or stripes) used to create the composite local time field. For instance a value of 8 means that 8 statistically processed fields have been used in the processing, each of them representing a section of 45 degrees of longitude (360/8) centered around the UTC time corresponding to the local time. (3) This is the number of forecasts and time steps used to create the statistically processed fields. These implicitly have the same statistical process as defined in octet 32. If a forecast has 2 time increments (3 hourly day 1 to 5 then 6 hourly), it should be encoded as 2 forecasts with the same reference time, using the appropriate starting forecast time and time increments. (4) This also represents the length of time range of the statistical processed fields. For instance, to create a 24h accumulation (encoded in octet 34-37), we could use several 3h accumulations, or 6h accumulation, a mixture of the two, etc.

TEMPLATE 4.98 Average, accumulation, extreme values or other statistically processed values of a post-processing individual ensemble forecast, control and perturbed, at a horizontal level or in a horizontal layer at a local Time

Octet No.	Contents
10	Parameter category (see Code table 4.1)
11	Parameter number (see Code table 4.2)
12-13	Input process identifier
14-15	Input originating centre
16	Type of post-processing
17	Type of generating process (see Code table 4.3)
18	Background generating process identifier (defined by originating centre)
19	Forecast generating process identifier (defined by originating centre)
20	Type of first fixed surface (see Code table 4.5)
21	Scale factor of first fixed surface
22–25	Scaled value of first fixed surface
26	Type of second fixed surface (see Code table 4.5)
27	Scale factor of second fixed surface
28–31	Scaled value of second fixed surface
32	Type of ensemble forecast (see Code table 4.6)
33	Perturbation number
34	Number of forecasts in ensemble
35	Statistical process used to calculate the fields that will be used in the local time processing (see Code table 4.10)
36	Indicator of unit of time for time range over which statistical processing is done. (see Code table 4.4)
37-40	Length of the time range over which statistical processing is done, in units defined by the previous octet. (see note 1)
41	Number of statistically processed fields used in the local time composite field (see Note 2)
42	Method used to calculate the field value at the local time specified in section 1 (see Code table 4.248)
43	n - number of forecasts used in the local time processing (see Note 3)
n/a	44-61 Specification of the first (or only) forecast
44-45	Year of the reference date for the first (or only) forecast
46	Month of the reference date for the first (or only) forecast
47	Day of the reference date for the first (or only) forecast
48	Hour of the reference time for the first (or only) forecast
49	Minute of the reference time for the first (or only) forecast
50	Second of the reference time for the first (or only) forecast
51	Indicator of units of forecast time (see Code table 4.4)
52-55	Forecast time in units defined by the previous octet
56	Number of increments for the successive forecast times
57	Indicator of unit of time for the time increment between the successive forecast times (see Code table 4.4)
58-61	Time increment between successive forecast times, in units defined by the previous octet (see Note 4)
n/a	62-nn These octets are included only if n>1 where nn=43+18*n
62-nn	(n-1) repetitions of sequence of octet 44-61, describing the next forecasts (reference date, reference time and forecast time increments)

Notes: (1) This represents the length of time over which the statistical processing was applied. The local time defined in section 1 represents the end of this processing. For instance, a value of 24h corresponds to a statistical processing between the previous day at local time and this day at local time. (2) This represents the number of statistically processed fields (or stripes) used to create the composite local time field. For instance a value of 8 means that 8 statistically processed fields have been used in the processing, each of them representing a section of 45 degrees of longitude (360/8) centered around the UTC time corresponding to the local time. (3) This is the number of forecasts and time steps used to create the statistically processed fields. These implicitly have the same statistical process as defined in octet 35. If a forecast has 2 time increments (3 hourly day 1 to 5 then 6 hourly), it should be encoded as 2 forecasts with the same reference time, using the appropriate starting forecast time and time increments. (4) This also represents the length of time range of the statistical processed fields. For instance, to create a 24h accumulation (encoded in octet 37-40), we could use several 3h accumulations, or 6h accumulation, a mixture of the two, etc.

[amilan17]

@sebvi  -- we have a new approach for managing notes for GRIB templates. I will present it at the next meeting, however, I can speak with you before then if needed.

[sebvi]

@amilan17 : saw it, it looks good. I see how you implemented it, should I created a noteID in the new table and then use that noteID in the relevant columnb of my templates?

[amilan17]

@sebvi , Yes exactly, but first review the existing notes and reference ones that are already established as applicable. For example, note 33 (Hours greater than 65534 will be coded as 65534.), is a common one and is already referenced 63 times in the templates.

[sebvi]

I have updated the proposal to move the local time method closer to the other keys it belongs to in proposed templates 4.95 to 4.98

I will provide sample GRIB either tomorrow or on Monday

[sebvi]

Having problem to upload as usual :S

[sebvi]

trying again templates.zip

[sebvi]

I have updated the branch.

@amilan17 During the process I noted that template 4.88 refers to table 2.248 instead of 4.248. Should I open a separate issue for an editorial change?

[amilan17]

@sebvi -- yes, please open a separate issue and tag me.  If we do this quickly, we can include in FT2021-2.

[sebvi]

@amilan17 I have already done so, it is issue #107

[jitsukoh]

@sebvi in the branch, I found misspelling in the name of PDT 4.95.

In GRIB2_CodeFlag_4_0_CodeTable_en.csv

Product definition template number,,95,,"Average, accumulation extreme values or other statiscally processed value at a horizontal level or in a horizontal layer at a specified local time",,,Operational

should be

Product definition template number,,95,,"Average, accumulation, extreme values or other statistically processed value at a horizontal level or in a horizontal layer at a specified local time",,,Operational

(added comma after accumulation and corrected the spelling of statistically)

And then all the Title_en in GRIB2_Template_4_95_ProductDefinitionTemplate_en.csv need to be corrected accordingly.

[amilan17]

@jitsukoh @sebvi 

I updated the branch

[jitsukoh]

@amilan17 thank you. I think the comma after "accumulation" in GRIB2_CodeFlag_4_0_CodeTable_en.csv is still missing.

[amilan17]

@sebvi  In the following templates, there are references to notes in the notes_en column, but no associated number in the noteIDs column

• GRIB2_Template_4_95_ProductDefinitionTemplate_en
• GRIB2_Template_4_96_ProductDefinitionTemplate_en
• GRIB2_Template_4_97_ProductDefinitionTemplate_en
• GRIB2_Template_4_98_ProductDefinitionTemplate_en

[amilan17]

@jitsukoh - thanks, I think I added all the necessary commas now.

[sebvi]

@amilan17 @jitsukoh I have updated the branch with whatever was missing, please have a look

[amilan17]

@sebvi -- Thank you very much for updating the notes. I merged your branch into FT21-2, but we are having difficulty verifying the content, because now the issue summary (at the top of this issue) does not match the content. Can you please update the issue summary? @chenxiaoxia2019 is double/triple checking everything and we need to get this done this week.

[amilan17]

Templates 4.93 and 4.94 in issue summary are inconsistent with content in FT21-2 branch.

[sebvi]

@amilan17 and @chenxiaoxia2019 : if you could point out what is not matching, that would help

[chenxiaoxia2019]

@sebvi Hi, Sebastien, it is about the added notes in Templates 4.93, 4.94. In the "Detailed Proposal", in these templates, there are no notes. Yet in the GitHub branch, there are notes.