Issues with some grib files, multi grib read, A-LAEF ensemble stepRange

[meteorolog90]

I have problem to read some grib files, so can't do verification with HARP.

Pressure at surface:

grib_ls SURFPRESSION_16.grb
SURFPRESSION_16.grb
edition      centre       typeOfLevel  level        dataDate     stepRange    shortName    packingType  gridType
1            217          heightAboveGround  0            20200726     16           pres         grid_simple  lambert
1 of 1 messages in SURFPRESSION_16.grb

1 of 1 total messages in 1 files

> pres <- read_grid("/users/ext005/HARP/SURFPRESSION_16.grb","ps")
Reading /users/ext005/HARP/SURFPRESSION_16.grb
Warning: Don't know how to read 'ps' from grib files.
Error: None of the requested parameters can be read from grib files.

Inca wind speed:

 grib_ls WS_INCA_1.grb
WS_INCA_1.grb
edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType
2            217          20200720     af           lambert      1            heightAboveGround  10           10si         grid_simple
1 of 1 messages in WS_INCA_1.grb

1 of 1 total messages in 1 files

> ws <-read_grid("/users/ext005/HARP/2020050400/WS_INCA_1.grb", "s10m")
Reading /users/ext005/HARP/2020050400/WS_INCA_1.grb
Warning: Parameter "s10m" (ws) not found in grib file.
Error: None of the requested data could be read from grib file: /users/ext005/HARP/2020050400/WS_INCA_1.grb

I had similar problems with tmax, tmin (attached). I use this list to define the parameters:

> show_harp_parameters()
# A tibble: 22 x 2
   harp_parameter_name description
   <chr>               <chr>
 1 AccPcp<X>h          Accumulated precipitation over <X> hours, e.g. AccPcp12h
 2 CChigh              High level cloud cover
 3 CClow               Low level cloud cover
 4 CCmed               Medium level cloud cover
 5 CCtot               Total cloud cover
 6 Cbase               Height of cloud base
 7 D10m                10m wind direction
 8 G10m                10m wind gust - period depends on input data
 9 Gmax                10m maximum wind gust - period depends on input data
10 Pcp                 Precipitation direct from model - usually accumulated fr~
11 Pmsl                Pressure at mean sea level
12 Ps                  Pressure at surface
13 Q2m                 2m specific humidity
14 RH2m                2m relative humidity
15 S10m                10m wind speed
16 Smax                Maximum 10m wind speed - period depends on input data
17 T2m                 2m temperature
18 Td2m                2m dewpoint temperature
19 Tmax                Maximum 2m temperature
20 Tmin                Minimum 2m temperature
21 Z                   Height above sea level
22 vis                 Horizontal visibility

Question number two: Is it possible to read multi grib files (using read_forecast function) for HARP verification ?

Inca wind speed multi grib structure:

WS_INCA.grb
edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType
2            217          20200720     af           lambert      0            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      1            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      2            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      3            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      4            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      5            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      6            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      7            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      8            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      9            heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      10           heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      11           heightAboveGround  10           10si         grid_simple
2            217          20200720     af           lambert      12           heightAboveGround  10           10si         grid_simple
13 of 13 messages in WS_INCA.grb

13 of 13 total messages in 1 files

A-LAEF ensemble grib structure( copy only one part, it is too long ):

A-LAEF_mem_16_2020080500_surface.grb
edition      centre       typeOfLevel  level        dataDate     stepRange    dataType     shortName    packingType  gridType     
1            ecmf         heightAboveGround  0            20200805     0            pf           t            grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           z            grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           lsp          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           cp           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           lssf         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           snoc         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           grad         grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     0            pf           10u          grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     0            pf           10v          grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     0            pf           2t           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           tcc          grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     0            pf           r            grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     0            pf           mx2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     0            pf           mn2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           ccc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           hcc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           mcc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           lcc          grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     0            pf           unknown      grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     0            pf           unknown      grid_simple  lambert     
1            ecmf         heightAboveSea  0            20200805     0            pf           msl          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0            pf           pres         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     1            pf           t            grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     1            pf           z            grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-1          pf           lsp          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-1          pf           cp           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-1          pf           lssf         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-1          pf           snoc         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-1          pf           grad         grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     1            pf           10u          grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     1            pf           10v          grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     1            pf           2t           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     1            pf           tcc          grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     1            pf           r            grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     0-1          pf           mx2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     0-1          pf           mn2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     1            pf           ccc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     1            pf           hcc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     1            pf           mcc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     1            pf           lcc          grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     0-1          pf           unknown      grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     0-1          pf           unknown      grid_simple  lambert     
1            ecmf         heightAboveSea  0            20200805     1            pf           msl          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     1            pf           pres         grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     0-1          pf           10fg         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-1          pf           tp           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-1          pf           sf           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     2            pf           t            grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     2            pf           z            grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-2          pf           lsp          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-2          pf           cp           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-2          pf           lssf         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-2          pf           snoc         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-2          pf           grad         grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     2            pf           10u          grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     2            pf           10v          grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     2            pf           2t           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     2            pf           tcc          grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     2            pf           r            grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     1-2          pf           mx2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     1-2          pf           mn2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     2            pf           ccc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     2            pf           hcc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     2            pf           mcc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     2            pf           lcc          grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     1-2          pf           unknown      grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     1-2          pf           unknown      grid_simple  lambert     
1            ecmf         heightAboveSea  0            20200805     2            pf           msl          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     2            pf           pres         grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     1-2          pf           10fg         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-2          pf           tp           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-2          pf           sf           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     3            pf           t            grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     3            pf           z            grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-3          pf           lsp          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-3          pf           cp           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-3          pf           lssf         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-3          pf           snoc         grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     0-3          pf           grad         grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     3            pf           10u          grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     3            pf           10v          grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     3            pf           2t           grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     3            pf           tcc          grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     3            pf           r            grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     2-3          pf           mx2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     2-3          pf           mn2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     3            pf           ccc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     3            pf           hcc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     3            pf           mcc          grid_simple  lambert     
1            ecmf         heightAboveGround  0            20200805     3            pf           lcc          grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     2-3          pf           unknown      grid_simple  lambert     
1            ecmf         heightAboveGround  10           20200805     2-3          pf           unknown      grid_simple  lambert     
1            ecmf         heightAboveSea  0            20200805     3            pf           msl          grid_simple  lambert  

i tried it, but can't loop over all stepRange length (reads only first step Range). I solved this problem by breaking it into several parts (using grib_api function grib_copy). Is there a better solution?

Question number three:

In this A-LAEF grib example, as we see, some parameters have two lead time intervals :

edition      centre       typeOfLevel  level        dataDate     stepRange    dataType     shortName    packingType  gridType     
1            ecmf         heightAboveGround  2            20200805     1-2          pf           mx2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     1-2          pf           mn2t6        grid_simple  lambert  
1            ecmf         heightAboveGround  2            20200805     2-3          pf           mx2t6        grid_simple  lambert     
1            ecmf         heightAboveGround  2            20200805     2-3          pf           mn2t6        grid_simple  lambert     

Is there a template to read such structured step range (1-2, 2-3, 3-4........)?

attached grib files grib.zip alaef_parts.zip - extract only part, too big for upload

Best regards, Martin

[andrew-MET]

30 fixes some if these issues:

1.

• You can now read surface pressure using "ps"
• INCA wind speed has a shortName of "10si" whereas harp expects it to be "ws". You can now specify your own parameter using `parameter = as_harp_parameter("10si")
• Tmin and Tmax are a bit unpredictable. I'm not sure what the 6 in the shortName means... in the last 6 hours I guess. Currently we have no way of easily expressing that in a generic parameter name so you can do the same as above and use parameter = as_harp_parameter("mx2t6") or similar for "mn2t6"
• Note I also see that you have ugst and vgst there. Since there is no implied level information in those parameters you would have to use as_harp_parameter("ugst", level_type = "height", level = 10)

2. In theory it should be possible to read grib files with multi-field messages by setting file_format_opts = grib_opts(multi = TRUE) However, that doesn't appear to work with Rgrib2 from @adeckmyn 's repository, but it does work with Rgrib2 from @harphub.

3. We don't actually use stepRange for the lead time, but rather the difference between the validDate:validTime and dataDate:dataTime. As long as the second of the two numbers in stepRange is different it's not a problem. However, if you had something like 1-3, 2-3, we can't deal with that at the moment.

[meteorolog90]

About multi grib When multi grib option is used, data is written into database but the warning messages is appears.

fcst_T2m <- read_forecast(

start_date=start,
end_date=end,
fcst_model="ALAEF",
parameter="T2m",
lead_time= seq(0,48,3),
members=seq(0,3,1),
by="12h",
file_path="/data/nwp/products/a-laef_stream",
file_template="A-LAEF_mem_{MBR2}_{YYYY}{MM}{DD}{HH}_surface.grb",
file_format_opts = grib_opts(multi = TRUE),
transformation="interpolate",
transformation_opts=interpolate_opts(stations=stanice,clim_file ="/users/ext005/HARP/2020050400/GEOPOTENCIAL.grb"),
output_file_opts=sqlite_opts(path="/work/users/ext005/sql/forecast/"),
return_data=TRUE,
show_progress =TRUE
)

Reading /users/ext005/HARP/2020050400/GEOPOTENCIAL.grb
Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2020081500_surface.grb
MULTI: found 1822 message(s) and 1822 field(s).
Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2020081500_surface.grb
MULTI: found 1822 message(s) and 1822 field(s).
Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2020081500_surface.grb
MULTI: found 1822 message(s) and 1822 field(s).
Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2020081500_surface.grb
MULTI: found 1822 message(s) and 1822 field(s).
Doing height correction for 2m temperature.
Opening connection to: /work/users/ext005/sql/forecast//ALAEF/2020/08/FCTABLE_T2m_202008_00.sqlite
Adding 'z' to index_cols.
Writing data

Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2020081512_surface.grb
MULTI: found 1822 message(s) and 1822 field(s).
Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2020081512_surface.grb
MULTI: found 1822 message(s) and 1822 field(s).
Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2020081512_surface.grb
MULTI: found 1822 message(s) and 1822 field(s).
Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2020081512_surface.grb
MULTI: found 1822 message(s) and 1822 field(s).
Doing height correction for 2m temperature.
Opening connection to: /work/users/ext005/sql/forecast//ALAEF/2020/08/FCTABLE_T2m_202008_12.sqlite
Adding 'z' to index_cols.
Writing data

There were 16 warnings (use warnings() to see them)

> warnings()
Warning messages:
1: Problem with `mutate()` input `forecast_data`.
x MULTI is still buggy!
i Input `forecast_data` is `purrr::map2(...)`.
i The error occurred in group 1: file_name = "/data/nwp/products/a-laef_stream/A-LAEF_mem_00_2020081500_surface.grb".
2: In Ginfo.character(filename, IntPar, DblPar, StrPar, rList = as.integer(1:nmessages),  ... :
  MULTI is still buggy!
3: Problem with `mutate()` input `forecast_data`.
x MULTI is still buggy!
i Input `forecast_data` is `purrr::map2(...)`.
i The error occurred in group 2: file_name = "/data/nwp/products/a-laef_stream/A-LAEF_mem_01_2020081500_surface.grb".
4: In Ginfo.character(filename, IntPar, DblPar, StrPar, rList = as.integer(1:nmessages),  ... :
  MULTI is still buggy!
5: Problem with `mutate()` input `forecast_data`.
x MULTI is still buggy!
i Input `forecast_data` is `purrr::map2(...)`.
i The error occurred in group 3: file_name = "/data/nwp/products/a-laef_stream/A-LAEF_mem_02_2020081500_surface.grb".
6: In Ginfo.character(filename, IntPar, DblPar, StrPar, rList = as.integer(1:nmessages),  ... :
  MULTI is still buggy!
7: Problem with `mutate()` input `forecast_data`.
x MULTI is still buggy!
i Input `forecast_data` is `purrr::map2(...)`.
i The error occurred in group 4: file_name = "/data/nwp/products/a-laef_stream/A-LAEF_mem_03_2020081500_surface.grb".
8: In Ginfo.character(filename, IntPar, DblPar, StrPar, rList = as.integer(1:nmessages),  ... :
  MULTI is still buggy!
9: Problem with `mutate()` input `forecast_data`.
x MULTI is still buggy!
i Input `forecast_data` is `purrr::map2(...)`.
i The error occurred in group 1: file_name = "/data/nwp/products/a-laef_stream/A-LAEF_mem_00_2020081512_surface.grb".
10: In Ginfo.character(filename, IntPar, DblPar, StrPar, rList = as.integer(1:nmessages),  ... :
  MULTI is still buggy!
11: Problem with `mutate()` input `forecast_data`.
x MULTI is still buggy!
i Input `forecast_data` is `purrr::map2(...)`.
i The error occurred in group 2: file_name = "/data/nwp/products/a-laef_stream/A-LAEF_mem_01_2020081512_surface.grb".
12: In Ginfo.character(filename, IntPar, DblPar, StrPar, rList = as.integer(1:nmessages),  ... :
  MULTI is still buggy!
13: Problem with `mutate()` input `forecast_data`.
x MULTI is still buggy!
i Input `forecast_data` is `purrr::map2(...)`.
i The error occurred in group 3: file_name = "/data/nwp/products/a-laef_stream/A-LAEF_mem_02_2020081512_surface.grb".
14: In Ginfo.character(filename, IntPar, DblPar, StrPar, rList = as.integer(1:nmessages),  ... :
  MULTI is still buggy!
15: Problem with `mutate()` input `forecast_data`.
x MULTI is still buggy!
i Input `forecast_data` is `purrr::map2(...)`.
i The error occurred in group 4: file_name = "/data/nwp/products/a-laef_stream/A-LAEF_mem_03_2020081512_surface.grb".
16: In Ginfo.character(filename, IntPar, DblPar, StrPar, rList = as.integer(1:nmessages),  ... :
  MULTI is still buggy!

When I look at the content written in the database, I do not see an error

[andrew-MET]

It's a warning coming from Rgrib2 to say that it might not be reliable for multi field grib messages. The rest is just traceback. I don't think you need to worry about it, but to be doubly sure it might be worth checking the values are what you expect.