search

harphub / harpIO

IO functions for HARP
https://harphub.github.io/harpIO/
Other
6 stars 16 forks source link

FA, HDF5 and GRIB fixes #40

Closed adeckmyn closed 3 years ago

adeckmyn commented 3 years ago

A lot of fixes for FA and HDF5 to have similar functionality as other read_x functions. Several fixes for grib-2.

andrew-MET commented 3 years ago

The "ws10si" is just from the error message, but it looks like it iterates over the loop of param_info$short_name twice, when it actually finds "ws" on the first pass, thus it doesn't find anything on the second pass when it's looking for "10si" and filters the data to zero rows

adeckmyn commented 3 years ago

I tested read_grid() with the grib files in harpData. I had no problems with "ws10m", "u10m". What gave the error for you? [edit] Oh I see. "S10m" gives an error while "ws10m" works. That's why I missed it. The break command needed to be 1 level lower. Fixed now.

adeckmyn commented 3 years ago

The first read_forecast() (from vignette) works for me. Did you run with latest fixes (S10m)? The second one with netcdf data fails. That indeed seems to happen in initialise_interpolation(). BUT: that routine shouldn't even run for netcdf, I believe. There even is a check for this at the start. But if the file_format is not explicitly passed, it isn't yet known at that point in the routine.

andrew-MET commented 3 years ago

Using this PR, and rebuilding the package, the first example does not run.

It fails in compare.geodomain.

packageVersion("meteogrid")
[1] ‘3.8.4.9001’

Presumably this PR requires a meteogrid version that is not on harphub and has the wrong minimum version requirement.

adeckmyn commented 3 years ago

Ah, that is probably correct. My own meteogrid version is 3.8.4.9006. I'll update meteogrid on harphup and increase the minimal version number.

adeckmyn commented 3 years ago

The problem wit the netcdf file was due to your changes in netcdf_opts(). Fixed now.

adeckmyn commented 3 years ago

Do you mean the line if (!is.null(file_format ) ** file_format %in% c("netcdf", "vfld")) { return(list(stations=stations)) } should not be there, or should not mention netcdf? I was under the impression that netcdf used its own interpolation code and needed no initialisation (except for the station list).

adeckmyn commented 3 years ago

For now, harpSpatial still calls initialise_interpolation() with a file name (for all the regridding to common grid). But that shouldn't do any harm.

andrew-MET commented 3 years ago

Entry in level_type column of output is grib code rather than a descriptive string. Can you fix it?

(Incidentally, the new Rgrib2 is not compatible with the current harpIO, only this PR. Hopefully this will only be a temporary issue that won't affect anyone as I think we're almost there in getting this PR merged)

Current version:

library(harpIO)
read_forecast(
  2018071000,
  2018071000,
  "AROME_Arctic",
  "T2m",
  file_path      = system.file("grib/AROME_Arctic", package = "harpData"),
  file_template  = "harmonie_grib_fp",
  return_data    = TRUE
)
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+000grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+003grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+006grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+009grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+012grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+015grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+018grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+021grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+024grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+027grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+030grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+033grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+036grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+039grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+042grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+045grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+048grib_fp
#> ● AROME_Arctic
#> # A tibble: 17 x 9
#>    fcdate              lead_time parameter validdate           level_type level
#>    <dttm>                  <dbl> <chr>     <dttm>              <chr>      <int>
#>  1 2018-07-10 00:00:00         0 T2m       2018-07-10 00:00:00 height         2
#>  2 2018-07-10 00:00:00         3 T2m       2018-07-10 03:00:00 height         2
#>  3 2018-07-10 00:00:00         6 T2m       2018-07-10 06:00:00 height         2
#>  4 2018-07-10 00:00:00         9 T2m       2018-07-10 09:00:00 height         2
#>  5 2018-07-10 00:00:00        12 T2m       2018-07-10 12:00:00 height         2
#>  6 2018-07-10 00:00:00        15 T2m       2018-07-10 15:00:00 height         2
#>  7 2018-07-10 00:00:00        18 T2m       2018-07-10 18:00:00 height         2
#>  8 2018-07-10 00:00:00        21 T2m       2018-07-10 21:00:00 height         2
#>  9 2018-07-10 00:00:00        24 T2m       2018-07-11 00:00:00 height         2
#> 10 2018-07-10 00:00:00        27 T2m       2018-07-11 03:00:00 height         2
#> 11 2018-07-10 00:00:00        30 T2m       2018-07-11 06:00:00 height         2
#> 12 2018-07-10 00:00:00        33 T2m       2018-07-11 09:00:00 height         2
#> 13 2018-07-10 00:00:00        36 T2m       2018-07-11 12:00:00 height         2
#> 14 2018-07-10 00:00:00        39 T2m       2018-07-11 15:00:00 height         2
#> 15 2018-07-10 00:00:00        42 T2m       2018-07-11 18:00:00 height         2
#> 16 2018-07-10 00:00:00        45 T2m       2018-07-11 21:00:00 height         2
#> 17 2018-07-10 00:00:00        48 T2m       2018-07-12 00:00:00 height         2
#> # … with 3 more variables: units <chr>, fcst_cycle <chr>,
#> #   AROME_Arctic_det <geolist>

PR version:

library(harpIO)
read_forecast(
  2018071000,
  2018071000,
  "AROME_Arctic",
  "T2m",
  file_path      = system.file("grib/AROME_Arctic", package = "harpData"),
  file_template  = "harmonie_grib_fp",
  return_data    = TRUE
)
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+000grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+003grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+006grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+009grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+012grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+015grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+018grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+021grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+024grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+027grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+030grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+033grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+036grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+039grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+042grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+045grib_fp
#> Reading /home/andrewts/R/x86_64-pc-linux-gnu-library/4.0/harpData/grib/AROME_Arctic/2018/07/10/00/fc2018071000+048grib_fp
#> ● AROME_Arctic
#> # A tibble: 17 x 9
#>    fcdate              lead_time parameter validdate           level_type level
#>    <dttm>                  <dbl> <chr>     <dttm>                   <int> <int>
#>  1 2018-07-10 00:00:00         0 T2m       2018-07-10 00:00:00        105     2
#>  2 2018-07-10 00:00:00         3 T2m       2018-07-10 03:00:00        105     2
#>  3 2018-07-10 00:00:00         6 T2m       2018-07-10 06:00:00        105     2
#>  4 2018-07-10 00:00:00         9 T2m       2018-07-10 09:00:00        105     2
#>  5 2018-07-10 00:00:00        12 T2m       2018-07-10 12:00:00        105     2
#>  6 2018-07-10 00:00:00        15 T2m       2018-07-10 15:00:00        105     2
#>  7 2018-07-10 00:00:00        18 T2m       2018-07-10 18:00:00        105     2
#>  8 2018-07-10 00:00:00        21 T2m       2018-07-10 21:00:00        105     2
#>  9 2018-07-10 00:00:00        24 T2m       2018-07-11 00:00:00        105     2
#> 10 2018-07-10 00:00:00        27 T2m       2018-07-11 03:00:00        105     2
#> 11 2018-07-10 00:00:00        30 T2m       2018-07-11 06:00:00        105     2
#> 12 2018-07-10 00:00:00        33 T2m       2018-07-11 09:00:00        105     2
#> 13 2018-07-10 00:00:00        36 T2m       2018-07-11 12:00:00        105     2
#> 14 2018-07-10 00:00:00        39 T2m       2018-07-11 15:00:00        105     2
#> 15 2018-07-10 00:00:00        42 T2m       2018-07-11 18:00:00        105     2
#> 16 2018-07-10 00:00:00        45 T2m       2018-07-11 21:00:00        105     2
#> 17 2018-07-10 00:00:00        48 T2m       2018-07-12 00:00:00        105     2
#> # … with 3 more variables: units <chr>, fcst_cycle <chr>,
#> #   AROME_Arctic_det <geolist>

Created on 2020-09-18 by the reprex package (v0.3.0)

This is important as the contents of this column is used to name the coluimn for vertical level when writing out to sqlite files in write_fctable_to_sqlite.R

col_name <- switch(
  level_type,
  "pressure" = "p",
  "model"    = "ml",
  "height"   = "z",
  "unknown"  = "level"
)
adeckmyn commented 3 years ago

Yes, Gindex() will confuse the current harpIO. But it was the only way. Hopefully only very short time.

I'll fix the level type.

In fact I wonder whether we could even use descriptive strings in level_type in get_grib_param_info. I am not sure whether there are string values that can be used directly in eccodes keys and which are consistent between grib-1 and grib-2. But that's not for this PR.

adeckmyn commented 3 years ago

Fixed. I can't remember why I modified this (in filter_grib_info()). Maybe I was confused. If there was a good reason, I forgot it again.

andrew-MET commented 3 years ago

@adeckmyn It looks like everything works as expected now, so unless you have anything more to add to the PR, I will merge tomorrow (22 Sep) afternoon.

adeckmyn commented 3 years ago

Nothing planned for now. So OK to merge.