Closed sebvi closed 1 year ago
25-April meeting: Sebastian to provide ecCodes version which encode/decodes GRIB2 messages
https://github.com/wmo-im/CCT/wiki/Teleconference.6.7.June.2023 notes:
samples are available; @sebvi update branch
Branch is updated
https://github.com/wmo-im/CCT/wiki/Teleconference.13.July.2023 notes:
Sebastien is responding to Anna's review in PR; @sebvi will provide samples and @SibylleK will do technical validation within a week; the FT proposal can be circulated to stakeholders/contributors during the NFP review;
an up-to-date eccodes is available here: eccodes
I have created some samples to download: WMO_tile.tar.zip
Branches look good with one question remaining in PR.
Branches look good with one question remaining in PR.
Can't find question. I assume I was mistaken!
@SibylleK to validate?
I tried to decode the provided example using the csv files of the branch.
As in the templates the "attribute of tile" should rather be a list of NUTAFTAC attributes, I am not satisfied with the notation. My suggestion would be: | Octet | Description |
---|---|---|
12 | Tile Classification (see code table 4.242) | |
13-14 | Type of Tile (see code table 4.251) | |
15 | Number of Used Spatial Tiles | |
16 | Number of Used Tile Attribute Combinations for Type of Tile | |
17 | Number of Used Tile Attributes for Tile Attribute Combination (NUTAFTAC) | |
. | Repeat the following octet for each attribute of tile (n=1,NUTAFTAC) | |
18+1(n-1) | List of attribute of tile (see code table 4.241) | |
.. | .. |
Regarding 4.116 - Individual ensemble forecast, control and perturbed on generalised tiles at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval the GRIB2 example and the proposal differ!
In the example the ensemble entries (i.e. Type of ensemble forecast, Perturbation number and Number of forecasts in ensemble) comes after the second fixed surface. In the proposal and the branch they are at the end of the template.
As in most (if not all) ensemble PDTs these entries come after the fixed surface types, I rather suppose changing the proposal.
With these changes I was able to read the entries in section 4 of the GRIB examples with a DWD GRIB reader software. The entries were the same as in an output of eccodes grib_dump. In that case the proposal has been validated.
@sebvi @amilan17 shall I update the branch?
dear @SibylleK , thank you for validating the templates. I an on annual leave at the moment but I will have a look at your comments later this evening.
Dear @SibylleK , I have now reviewed your comments and I fully agree with you.
I have updated the proposal and will update the branch tomorrow morning
I have updated the branch
@SibylleK -- Did you want to check this again or shall we consider it validated?
@amilan17, I double checked and the updated branch is OK. It can be considered validated.
Initial request
Several templates for encoding land surface tiling schemes have been proposed in the past and can be divided into 2 categories:
From the annex describing the spatio-temporal changing tiles, it is clear that each grid box contains N fixed number of tiles and that the tile types are different from grid box to grid box: with N=3, one grid box could be ocean/forest/crop and another grid box could be urban/lake/bare-rock. The tiles are labelled 1 to N with tile 1 being the highest fraction, tile 2 the next highest and so on to tile N with the smallest fraction. It implies that tile 1 in one grid box could be for instance "ocean" but it could be "deciduous forest" in another grid box. Hence the data is encoded over a set of GRIB2 messages that need to work together to recover what "tile 1" is in a given grid box: typically one message containing the "tile class" parameter using parameter encoded in discipline 2 category 0 entry 35 (2/0/35), which points to Code Table 4.243 and one message to encode the fraction or percentage using 2/0/36 and 2/0/37 respectively.
These templates have a number of caveats, practical and technical:
Our use case at ECMWF is that now we now want to encode traditional land surface parameters (2m temperature, 10 metre winds, humidity, you name it...) per specific type of tiles: 2m temp of urban areas, over lake, over ocean. This is very relevant because these parameters can be very different over different land surfaces within a grid box and this is usually a source of apparent error when only the spatial average temperature of the grid box is given. With the existing spatio-temporal tile approach, encoding such parameters is possible but is extremely cumbersome as it can't be encoded in a single GRIB message.
We are therefore proposing a new set of tile templates building on the existing spatio-temporal tile approach, trying to fix the issues listed above: The type of tiles is no longer a set of separate GRIB messages listing the types by decreasing importance, defined through a tile class parameter. Instead the type of tile is included in the template itself, avoiding the need for the extra set of messages. In this approach, tile 1 is always the same type across all grid boxes, and the same for tile 2 to N. We are also introducing a unique identifier to easily match parameters belonging to the same tile of the same model run configuration.
To prepare these templates, we have been in contact with several land surface modelling groups from ECMWF (IFS), MeteoFrance/SMHI (surfex model) and DWD (ICON) to make sure the new set of templates would fit all use cases. We are proposing a new table listing land type according to the major land cover surveys defined in existing Code Table 2.242 (which we are also extending). The new code table 4.251 is organised to group land type by surveys and can be extended to accommodate new future surveys. Each survey range is divided in a range for base land types and a range for composed land types. This is usually because land surface models tend to group native cover into major groups. A good example of this is how the various vegetation types can be grouped. Note that the dominant tiles approach can still be used if the tile type is set to 3501 and the parameter 2/0/35 is provided again for the tiles 1 to N.
Octet 12: defines the Land survey used to derive the type of tiles. The surveys are listed in the existing Code Table 4.242 and we are proposing to add 3 more surveys to this table. Octets 13-14: define the type of tile within a specified survey in octet 12. Code table 4.251 is new. We did not want to reuse Code table 4.243 which contains a subset of Code Table 2.251 because 4.243 is tightly couple to its use with the parameter "tile class", discipline 2, category 0, entry 35. Code Table 4.251 is organised in the following way: entries 1-1000 are for generic tile type not coupled with a particular survey and are meant to be used for inter-comparison. Then each subsequent range of a thousand are assigned to a specific Land survey. Within that each range, the first half is for "base tile type" of that survey while the second half is used for groups of tiles, either because the model does not fully resolved the base tiles and because a model could output parameters averaged over a specific group of tiles. For instance, a model could use the entries 1007-1015 for the various types of forest covers or only use the entry 1503 which is the grouping of all the forest covers. Octet 15: defines the number of tile types without considering additional attributes, i.e. the number of different value that Octets 13-14 could take in the given setup. Octet 16: defines the number of combination of type attributes for the type of tile given in Octets 13-14. For instance, if for the tile type "forest" there exit the following combination, "bare forest", "Forest + intercepted snow", "Forest + flooded" and "forest + intercepted snow + flooded", then octet 16 should be set to 4. This implies that 4 separate GRIB message will be provided, one for each combination. octet 17: defines NUTAFTAC which is the number of attributes specified in the following octet(s) 18 to 18 + NUTAFTAC. octect(s) 18 to 18 + NUTAFTAC-1: if NUTAFTAC=1, then only octet 18 is used to encode 1 attribute, if NUTAFTAC=2, then octects 18 and 19 are used to encode 2 attributes, etc. octet 19 + NUTAFTAC-1: sum of all combination of type of tiles with their attributes. For instance if a model uses 2 types of tile and type 1 has octet 16 set to 2 and type 2 has octet 16 set to 3, then this octet should be set to 2+3 = 5 octet 20 + NUTAFTAC-1: this is an index given to identify a particular combination of the previous tile keys. In principle, it ranges between 1 and the value encoded in octet 20+NUTAFTAC-1 octets 21 + (NUTAFTAC-1) - 36 + (NUTAFTAC-1) : 16 octets used to encode a unique identifier that encodes the specific tile configuration. All GRIB message sharing the same UUID are to be interpreted together. This provides a validation that the independent GRIB messages belong to the same run.
Amendment details
Comments
No response
Requestor(s)
Matthew Griffith (ECMWF) Sebastien Villaume (ECMWF)
Stakeholder(s)
ECMWF, Meteo France, SMHI, DMI, FMI, Met Norway and DWD
Publication(s)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB section 4, product definition template number 113 (new) Manual on Codes (WMO-No. 306), Volume I.2, GRIB section 4, product definition template number 114 (new) Manual on Codes (WMO-No. 306), Volume I.2, GRIB section 4, product definition template number 115 (new) Manual on Codes (WMO-No. 306), Volume I.2, GRIB section 4, product definition template number 116 (new) Manual on Codes (WMO-No. 306), Volume I.2, Code Table 4.251 (new) Manual on Codes (WMO-No. 306), Volume I.2, Code Table 4.0 (add entries) Manual on Codes (WMO-No. 306), Volume I.2, Code Table 4.241 (add entries) Manual on Codes (WMO-No. 306), Volume I.2, Code Table 4.242 (add entries)
Expected impact of change
MEDIUM
Collaborators
ECMWF, MeteoFrance, SMHI, DWD
References
No response
Validation
We can provide an ecCodes version which encode/decodes GRIB2 messages making use of the features described in this proposal