new templates for probability forecasts based on focal statistics

sebvi commented 1 year ago

Initial request

This proposal extends the probability templates 4.119 and 4.120 by additional keys which allow to encode a spatiotemporal processing based on focal (moving windows) statistics. The design of the template is inspired by the focal statistics functionality available in the ArcGIS software, documented under https://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-focal-statistics-works.htm .

The templates are intended to be used to encode probability products of upscaled high-resolution forecasts. For example, probabilities of precipitation based on ensemble forecasts of high spatial resolution show often low point probabilities as the fine structures in the forecasts are located at different positions in the ensemble members. With the upscaling, the spatial resolution of the original forecast is the same, but for each individual grid cell, a new value is assigned, which is based on a statistical processing of values in the grid cells neighbourhood.

Figure 1: Example of focal statistics with the maximum function

Figure 2: Example of an upscaled forecast using focal statistics with maximum function (https://doi.org/10.1002/met.1674) Exceedance probabilities are then calculated based on the upscaled raster’s. a) raw forecast of echotop18 b) Upscaled with maximum function and 9x9 grid cells rectangle

Another use case is the encoding of strike probabilities which are based on a spatiotemporal proximity criterion.

Figure 3: Example for the calulation of hurricane strike probabilities based on hurricane tracks.

A branch with an implementation of this template is accessible under: https://github.com/ecmwf/eccodes/tree/feature/ECC-1705-probabilitiesWithFocalStats

Amendment details

ADD the following templates entries in code table 4.0.

Code	Name
121	Probability forecasts from large ensembles with spatiotemporal processing based on focal (moving window) statistics at a horizontal level or in a horizontal layer at a point in time
122	Probability forecasts from large ensembles with spatiotemporal processing based on focal (moving window) statistics at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

ADD a new Code Table 4.103 - Spatial vicinity type.

Code	Name
0	Circle [m] (see note)
1	Rectangle [m, m] (see note)
2	Square [m] (see note)
3	Wedge [m, degree, degree] (see note)
4	Number of grid cells [-, -]
5-191	Reserved
192-254	Reserved for local use
255	Missing

Notes: The following additional arguments must be specified: • Circle - 1 argument for the radius in metres • Rectangle - 2 arguments for the length in 1. west-east and 2. south-north in metres • Square - 1 argument for the length of the equal-length sides in metres • Wedge - 3 arguments for 1. radius in metres and 2. start and 3. end radius in arithmetic degrees with 0 on the positive axis along west-east and counted counter-clockwise. • Number of grid cells - 2 arguments regarding the number of grid cells in 1. west-east and 2. south-north

ADD a new Code Table 4.104 - Spatial and temporal vicinity processing.

Code	Name
0	Average
1	Reserved
2	Maximum
3	Minimum
4	Range
5	Reserved
6	Standard deviation
7	Reserved
8	Reserved
11	Sum
12-189	Reserved
190	Quantile (see note)
191	Categorical (binary)
192-254	Reserved for local use
255	Missing

Note: The option quantile needs two additional arguments, 1. The total number of quantiles and 2. the quantile value. Compare templates 4.86 or 4.87.

ADD a new Code Table 4.105 - Spatial and temporal vicinity missing data.

Code	Name
0	Ignore missing data
1	No Data

Add to code table 4.2 new entries to Product discipline 0 - Meteorological products, parameter category 191: miscellaneous.

Parameter number	name	Units
5	Hurricane track in spatiotemporal vicinity	boolean
6	Tropical Storm track in spatiotemporal vicinity	boolean
7	Tropical Depression track in spatiotemporal vicinity	boolean

Note. 0/191/5-7 are intended to be used with templates 4.121 and 4.122 in which the spatiotemporal criteria is encoded used to get a categorical yes/no per grid point of each ensemble member.

Add to code table 4.9

Code	Name
9	Probability based on counts of categorical yes/no per member

Note: For Code no 9, Scale Factor of Lower Limit, Scaled Value of Lower Limit, Scale Factor of Upper Limit and Scaled Value of Upper Limit must be set to missing. This entry can be used with 0/191/5-7 entries but are not limited to them.

ADD Template 4.121 - Probability forecasts from large ensembles with spatiotemporal processing based on focal (moving window) statistics at a horizontal level or in a horizontal layer at a point in time.

Octet	Number of octets	Description
10	1	Parameter Category (see code table 4.1)
11	1	Parameter Number (see code table 4.2)
12	1	Type of Generating Process (see code table 4.3)
13	1	Background Process
14	1	Generating Process Identifier
15-16	2	Hours After Data Cut-off
17	1	Minutes After Data Cut-off
18	1	Indicator of Unit of Time Range (see code table 4.4)
19-22	4	Forecast Time
23	1	Type of First Fixed Surface (see code table 4.5)
24	1	Scale Factor of First Fixed Surface
25-28	4	Scaled Value of First Fixed Surface
29	1	Type of Second Fixed Surface (see code table 4.5)
30	1	Scale Factor of Second Fixed Surface
31-34	4	Scaled Value of Second Fixed Surface
35	1	Type of ensemble forecast (see Code table 4.6)
36-39	4	Number of forecasts in ensemble
40	1	Forecast Probability Number
41	1	Total Number of Forecast Probabilities
42	1	Probability Type (see code table 4.9)
43	1	Scale Factor of Lower Limit
44-47	4	Scaled Value of Lower Limit
48	1	Scale Factor of Upper Limit
49-52	4	Scaled Value of Upper Limit
53	1	Spatial Vicinity Type (see code table 4.103)
54	1	Number of Spatial Vicinity Values – NSV
		The next entry repeats nsv=1:NSV times
(55+(nsv-1)4) – (58+(nsv-1)4)	4	Spatial Vicinity Value
59+(nsv-1)*4	1	Spatial Vicinity Processing (see code table 4.104)
(60+(nsv-1)4)-(61+(nsv-1)4)	2	Spatial Vicinity Processing Argument 1 (e.g., if previous is quantile)
(62+(nsv-1)4)-(63+(nsv-1)4)	2	Spatial Vicinity Processing Argument 2 (e.g., if previous is quantile)
64+(nsv-1)*4	1	Spatial Vicinity Missing Data (see code table 4.105)
(65+(nsv-1)*4)	1	Temporal Vicinity Processing (see code table 4.104)
(66+(nsv-1)*4)	1	Temporal Vicinity Unit (see code table 4.4)
(67+(nsv-1)4)-(70+(nsv-1)4)	4	Temporal Vicinity Towards Past
(71+(nsv-1)4)-(74+(nsv-1)4)	4	Temporal Vicinity Towards Future

ADD Template 4.122 - Probability forecasts with spatiotemporal processing based on focal (moving window) statistics at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval.

Octet	Number of octets	Description
10	1	Parameter Category (see code table 4.1)
11	1	Parameter Number (see code table 4.2)
12	1	Type of Generating Process (see code table 4.3)
13	1	Background Process
14	1	Generating Process Identifier
15-16	2	Hours After Data Cut-off
17	1	Minutes After Data Cut-off
18	1	Indicator of Unit of Time Range (see code table 4.4)
19-22	4	Forecast Time
23	1	Type of First Fixed Surface (see code table 4.5)
24	1	Scale Factor of First Fixed Surface
25-28	4	Scaled Value of First Fixed Surface
29	1	Type of Second Fixed Surface (see code table 4.5)
30	1	Scale Factor of Second Fixed Surface
31-34	4	Scaled Value of Second Fixed Surface
35	1	Type of ensemble forecast (see Code table 4.6)
36-39	4	Number of forecasts in ensemble
40	1	Forecast Probability Number
41	1	Total Number of Forecast Probabilities
42	1	Probability Type (see code table 4.9)
43	1	Scale Factor of Lower Limit
44-47	4	Scaled Value of Lower Limit
48	1	Scale Factor of Upper Limit
49-52	4	Scaled Value of Upper Limit
53-54	2	Year of end of overall time interval
55	1	Month of end of overall time interval
56	1	Day of end of overall time interval
57	1	Hour of end of overall time interval
58	1	Minute of end of overall time interval
59	1	Second of end of overall time interval
60	1	n - number of time range specifications describing the time intervals used to calculate the statistically processed field
61-64	4	Total number of data values missing in statistical process
		65+(nsv-1)4) - 76+(nsv-1)4) Specification of the outermost (or only) time range over which statistical processing is done
65	1	Statistical process used to calculate the processed field from the field at each time increment during the time range (see code table 4.10)
66	1	Type of time increment between successive fields used in the statistical processing (see code table 4.11)
67	1	Indicator of unit of time for time range over which statistical processing is done (see code table 4.4)
68-71	4	Length of the time range over which statistical processing is done, in units defined by the previous octet
72	1	Indicator of unit of time for the increment between the successive fields used (see code table 4.4)
73-76	4	Time increment between successive fields, in units defined by the previous octet
		77 - nn These octets are included only if n > 1, where nn = 64 + 12 x n
77-88	12	As octets 65-76, next innermost step of processing
89-nn	n/a	Additional time range specifications, included in accordance with the value of n. Contents as octets 65-76, repeated as necessary nn = 64 + 12 x n
nn + 1	1	Spatial Vicinity Type (see code table 4.103)
nn + 2	1	Number of Spatial Vicinity Values – NSV
		The next entry repeats nsv=1:NSV times
(nn + 3+(nsv-1)4)–(nn + 6+(nsv-1)4)	4	Spatial Vicinity Value
nn + 7+(nsv-1)*4	1	Spatial Vicinity Processing (see code table 4.104)
(nn + 8+(nsv-1)4)-(nn + 9+(nsv-1)4)	2	Spatial Vicinity Processing Argument 1 (e.g., if previous is quantile)
(nn + 10+(nsv-1)4)-(nn + 11+(nsv-1)4)	2	Spatial Vicinity Processing Argument 2 (e.g., if previous is quantile)
nn + 12+(nsv-1)*4	1	Spatial Vicinity Missing Data (see code table 4.105)
(nn + 13+(nsv-1)*4)	1	Temporal Vicinity Processing (see code table 4.104)
(nn + 14+(nsv-1)*4)	1	Temporal Vicinity Unit (see code table 4.4)
(nn + 15+(nsv-1)4)-(nn + 18+(nsv-1)4)	4	Temporal Vicinity Towards Past
(nn + 19+(nsv-1)4)-(nn + 22+(nsv-1)4)	4	Temporal Vicinity Towards Future

Comments

No response

Requestor(s)

Robert Osinski (ECMWF) Sebastien Villaume (ECMWF)

Stakeholder(s)

ECMWF

Publication(s)

Example: Manual on Codes (WMO-No. 306), Volume I.2, GRIB templates + table in section 4

Expected impact of change

MEDIUM

Collaborators

No response

References

No response

Validation

No response

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.17.18.October.2023 notes: Sebastien introduced the proposal; team has no questions yet

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.21.22.November.2023 notes: @sebvi add note on the units; Sebastien will update the branch and prepare samples;

amilan17 commented 1 year ago

https://github.com/wmo-im/CCT/wiki/Teleconference.21.29.November.2023 notes:

in progress; add images as appendix with proper credit if needed;

sebvi commented 11 months ago

branch updated

sebvi commented 10 months ago

Adding the appendix to explain better the concepts behind focal statistics. Thanks to @rdosinski for preparing the appendix Appendix_FocalStatsProposals.docx

sebvi commented 10 months ago

I am attaching a sample file together with the original field be fore applying the focal statistics

GRIB2_225.zip

amilan17 commented 10 months ago

https://github.com/wmo-im/CCT/wiki/Teleconference.10.January.2024 notes:

Sebastien added a doc for an appendix; @sebvi will update the proposal; @SibylleK to validate the sample; @amilan17 revalidate the branch

SibylleK commented 10 months ago

I tried to validate the sample (only from a purely technical point of view). The entries in section 4 of the GRIB example file PDTN_121_upscaled_tprate.grib2 were read with a DWD GRIB reader software and the entries were the same as in an output of eccodes grib_dump.

Here are some comments:

Regarding the "Spatial Vicinity Value", I would propose to name it "List of spatial vicinity values" in both templates, as the entry should be repeated NSV times.
In the csv-Files I stumbled across some "EN_DASH" (UTF-8(hex) = e28093) instead of a normal "HYPHEN-MINUS" (UTF-8(hex) = 2d) :
1. at "54,Number of Spatial Vicinity Values – NSV" and
2. "(55+(nsv-1)4) – (58+(nsv-1)4)" in GRIB2_Template_4_121_ProductDefinitionTemplate_en.csv
3. at "nn + 2,Number of Spatial Vicinity Values – NSV" and
4. "(nn + 3+(nsv-1)4)–(nn + 6+(nsv-1)4)" in GRIB2_Template_4_122_ProductDefinitionTemplate_en.csv

@amilan17, @sebvi , I hope it is not to late to make these minor changes.

Apart from that, from a technical point of view, the proposal can be considered validated.

wmo-im / GRIB2