search

harphub / harpIO

IO functions for HARP
https://harphub.github.io/harpIO/
Other
6 stars 17 forks source link

read_eps_interpolate - Can't join on 'member' x 'member' #21

Closed meteorolog90 closed 4 years ago

meteorolog90 commented 4 years ago

I trying to read grib :

A-LAEF_mem_01_2020050400_1_surface.grb A-LAEF_mem_01_2020050400_2_surface.grb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-LAEF_mem_02_2020050412_1_surface.grb

date: 2020-05-04 time interval 00,12 members: 00,01,02,03 stepRange: 0,1,2,3,4

and i got this error:

here() starts at /users/ext005/HARP Loading required package: shiny Initialising interpolation. Reading data from /users/ext005/HARP/SURFGEOPOTENTIEL_00.grb. Generating file names. Reading data for 202005040000 Date assumed to be YYYYMMDDHHmm Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_00_2020050400_0_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_01_2020050400_0_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_02_2020050400_0_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_03_2020050400_0_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_00_2020050400_1_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_01_2020050400_1_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_02_2020050400_1_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_03_2020050400_1_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_00_2020050400_2_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_01_2020050400_2_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_02_2020050400_2_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_03_2020050400_2_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_00_2020050400_3_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_01_2020050400_3_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_02_2020050400_3_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_03_2020050400_3_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_00_2020050400_4_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_01_2020050400_4_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_02_2020050400_4_surface.grb. Reading data from /users/ext005/HARP/2020050400/format_test/A-LAEF_mem_03_2020050400_4_surface.grb. Joining data. Error: Can't join on 'member' x 'member' because of incompatible types (character / integer) Execution halted

Problem is for 12h interval

My code:

fcst <- read_eps_interpolate(

  start_date = 2020050400,
  end_date =   2020050412,
  station = stanice,
  eps_model = "A-LAEF",
  parameter = "T2m",
  lead_time = seq(0,4,1),
  members_in = seq(0,3,1),
  by="12h",
  file_path=here("2020050400/format_test"),
  file_format = "grib",
  file_template = "{eps_model}_mem_{MBR2}_{YYYY}{MM}{DD}{HH}_{LDT}_surface.grb",
  clim_file = here("SURFGEOPOTENTIEL_00.grb"),
  sqlite_path = here(),
  sqlite_template= "fctable_eps",
  return_data=TRUE
  )

Harp version: '0.0.0.9149' tidyverse: '1.3.0'

andrew-MET commented 4 years ago

Unfortunately read_eps_interpolate has had a lot of problems, especially in dealing with grib files from different centres, and we've decided to rethink the whole thing and have replaced it with a new function, read_forecast (see https://harphub.github.io/harpIO/articles/read_raw_forecast.html and https://harphub.github.io/harpIO/articles/transformations.html).

For your particular application you will need to do the following:

fcst <- read_forecast(
  start_date          = 2020050400,
  end_date            = 2020050412,
  fcst_model          = "A-LAEF",
  parameter           = "T2m",
  lead_time           = seq(0,4),
  members             = seq(0,3),
  by                  = "12h",
  file_path           = here("2020050400/format_test"),
  file_template       = "{fcst_model}_mem_{MBR2}_{YYYY}{MM}{DD}{HH}_{LDT}_surface.grb",
  transformation      = "interpolate",
  transformation_opts = interpolate_opts(stations = stanice, clim_file = here("SURFGEOPOTENTIEL_00.grb"))
  output_file_opts    = sqlite_opts(path = here()),
  return_data         = TRUE
)

The main differences are that eps_model is now fcst_model, members_in is now members, interpolation is now set through the transformation and transformation_opts arguments, with interpolate_opts used to generate the options for the interpolation (here you only set the station list and the clim_file and keep everything else as the default), and the output file is set with output_file_opts and sqlite_opts.

You probably need to update your version of harpIO and hopefully this should work for you.

Soon there will be a message in read_eps_interpolate recommending to use read_forecast instead.

meteorolog90 commented 4 years ago

Update HARP( version 0.0.0.9157) but problem not disappear. When use this code: 

fcst <- read_forecast(
  start_date          = 2020050400,
  end_date            = 2020050412,
  fcst_model          = "A-LAEF",
  parameter           = "T2m",
  lead_time           = seq(0,4),
  members             = seq(0,3),
  by                  = "12h",
  file_path           = here("2020050400/format_test"),
  file_template       = "{fcst_model}_mem_{MBR2}_{YYYY}{MM}{DD}{HH}_{LDT}_surface.grb",
  transformation      = "interpolate",
  transformation_opts = interpolate_opts(stations = stanice, clim_file = here("SURFGEOPOTENTIEL_00.grb"))
  output_file_opts    = sqlite_opts(path = here()),
  return_data         = TRUE
)

i got this error :

_Reading /users/ext005/HARP/SURFGEOPOTENTIEL_00.grb Error : 'Gindex' is not an exported object from 'namespace:Rgrib2' Error: Cannot get 'sfc_geo' from: /users/ext005/HARP/SURFGEOPOTENTIEL_00.grb. Maybe you need to set the correct 'clim_param' with: transformation_opts = interpolate_opts(climparam = ) Execution halted

Test with harpData:

read_forecast(
  2018071000,
  2018071000,
  "AROME_Arctic",
  "S10m",
  transformation = "interpolate",
  file_path      = system.file("grib/AROME_Arctic", package = "harpData"),
  file_template  = "harmonie_grib_fp",
  return_data    = TRUE
)

throw:

_Warning: 'transformation_opts' not set for transformation = 'interpolate'. Using default interpolate_opts() No stations specified. Using default stations: 'station_list' Initializing interpolate weights using 'sfc_geo'. Reading /users/nwp111/R/powerpc64/hpccmp10/3.6/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+000grib_fp Error : 'Gindex' is not an exported object from 'namespace:Rgrib2' 'sfc_geo' not found. Attempting to get domain from file. Reading /users/nwp111/R/powerpc64/hpccmp10/3.6/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+000gribfp Error: 'Gindex' is not an exported object from 'namespace:Rgrib2' Execution halted

meteorolog90 commented 4 years ago

I was reinstall HARP using this packages, and now it working: devtools::install_github("adeckmyn/Rgrib2") remotes::install_github("andrew-MET/harpIO") remotes::install_github("andrew-MET/harpPoint") remotes::install_github("andrew-MET/harpData") remotes::install_github("https://github.com/adeckmyn/meteogrid.git")

_>data <- read_grid(system.file("grib/HARMUK20171015T12Z+003.grib", package = "harpData"),parameter = "T2m") Reading /users/ext005/renv/test1/renv/library/R-3.6/powerpc64-unknown-linux-gnu/harpData/grib/HARMUK20171015T12Z+003.grib

_

With this packages it throw "Gindex" error: remotes::install_github("harphub/harpIO") remotes::install_github("harphub/harpData") remotes::install_github("harphub/meteogrid") remotes::install_github("harphub/Rgrib2")

_> data <- readgrid(system.file("grib/HARMUK20171015T12Z+003.grib", package = "harpData"),parameter = "T2m") Reading /users/ext005/renv/test/renv/library/R-3.6/powerpc64-unknown-linux-gnu/harpData/grib/HARMUK20171015T12Z+003.grib Error: 'Gindex' is not an exported object from 'namespace:Rgrib2'

andrew-MET commented 4 years ago

This is fixed now. Alex's version of Rgrib2 includes an enhancement to speed up grib reading. I implemented it in harp just before going on holiday, but forgot to add a check for the Rgrib2 version - this is now done (https://github.com/harphub/harpIO/pull/23/commits/6225e2f1ff05209a9a5b01ec02371339694a0ed4). If you stick with the "adeckmyn" version of Rgrib2 you should benefit from faster reading of grib files, particularly if the grib files contain a large a number of messages. This version of Rgrib2 will likely be on harphub after the summer.