search

harphub / harpIO

IO functions for HARP
https://harphub.github.io/harpIO/
Other
6 stars 16 forks source link

A-LAEF read parameters #77

Closed meteorolog90 closed 3 years ago

meteorolog90 commented 3 years ago

We have problem to read tp parameter from A-LAEF gribs. 

station <- read.csv("/users/ext005/HARP/STANICE.csv")
startDate = 2021040600;
endDate =   2021040712;

forecast <- read_forecast(

start_date            = startDate, 
  end_date            = endDate,
  fcst_model          = "alaef_cy40_5km",
  parameter           = "Pcp",
  lead_time           = seq(0,3,1),
  members             = seq(0,3,1),
  by                  = "12h",
  file_path           = "/data/nwp/products/a-laef_stream",
  file_template       = "A-LAEF_mem_{MBR2}_{YYYY}{MM}{DD}{HH}_surface.grb",
  file_format_opts    = grib_opts(
                                  multi=TRUE,
                                  param_find = list(Pcp = use_grib_paramId(228)),
                                  level_find = list(Pcp = use_grib_indicatorOfTypeOfLevel(105, 0))
                                  ),
  transformation      = "interpolate",
  transformation_opts = interpolate_opts(stations=station),
  output_file_opts=sqlite_opts(path="/work/users/ext005/sql/forecast/test"),                                
  return_data         = TRUE,
  show_progress       = TRUE
)

Initializing interpolate weights using 'sfc_geo'. Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040600_surface.grb More than 2000 GRIB messages! 'sfc_geo' not found. Attempting to get domain from file. Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040600_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2021040600_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2021040600_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2021040600_surface.grb More than 2000 GRIB messages! Warning: No data found for 202104060000. Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040612_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2021040612_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2021040612_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2021040612_surface.grb More than 2000 GRIB messages! Warning: No data found for 202104061200. Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040700_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2021040700_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2021040700_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2021040700_surface.grb More than 2000 GRIB messages! Warning: No data found for 202104070000. Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040712_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2021040712_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2021040712_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2021040712_surface.grb More than 2000 GRIB messages! Warning: No data found for 202104071200. Warning message: There were problems reading: /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040600_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2021040600_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2021040600_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2021040600_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040612_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2021040612_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2021040612_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2021040612_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040700_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2021040700_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2021040700_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2021040700_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040712_surface.grb /data/nwp/products/a-laef_stream/A-LAEF_mem01 [... truncated]

I guess the problem is in the stepRange formatting.

alaefTP

Any suggestion for this problem ?

andrew-MET commented 3 years ago

We don't use stepRange, so it can't be that.

I see that you are using use_grib_paramId - I don't think that's a key for grib 1 data, and the edition column says it's grib 1. You would need use_grib_indicatorOfParameter.

However, I think parameter = "Pcp" should work without any explicit options as it looks for shortName = "tp" and typeOfLevel = "heightAboveGround". What happens if you remove the whole file_format_opts argument?

meteorolog90 commented 3 years ago

hmm, removing format arg fix this...... So in the new version, we don't have to define multi=TRUE for multi grib. Nice :)

forecast <- read_forecast(

start_date            = startDate, 
  end_date            = endDate,
  fcst_model          = "alaef_cy40_5km",
  parameter           = "Pcp",
  lead_time           = seq(0,3,1),
  members             = seq(0,3,1),
  by                  = "12h",
  file_path           = "/data/nwp/products/a-laef_stream",
  file_template       = "A-LAEF_mem_{MBR2}_{YYYY}{MM}{DD}{HH}_surface.grb",
  transformation      = "interpolate",
  transformation_opts = interpolate_opts(stations=station),
  output_file_opts=sqlite_opts(path="/work/users/ext005/sql/forecast/test"),                                
  return_data         = TRUE,
  show_progress       = TRUE
)

Initializing interpolate weights using 'sfc_geo'. Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040600_surface.grb More than 2000 GRIB messages! 'sfc_geo' not found. Attempting to get domain from file. Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_00_2021040600_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_01_2021040600_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_02_2021040600_surface.grb More than 2000 GRIB messages! Reading /data/nwp/products/a-laef_stream/A-LAEF_mem_03_2021040600_surface.grb More than 2000 GRIB messages! Opening connection to: /work/users/ext005/sql/forecast/test/alaef_cy40_5km/2021/04/FCTABLE_Pcp_202104_00.sqlite Adding 'z' to index_cols. Writing data

...... ...... ......

andrew-MET commented 3 years ago

multi = TRUE is just for grib 2 and should be used when there are multiple fields in a single message. It doesn't have a purpose for grib 1.