vfld files without TEMP section is not handled properly

[uandrae]

Presenting a vfld file with out a temp section e.g. 1 synop station and 0 temp stations

  1     0     4
          14
 FI               0
 NN               0
     01001   70.933   -8.667    66.7   1.0000E+00
         12
           1
 PP           0

makes the reading crash with

Reading data for 202009300000
Date assumed to be YYYYMMDDHHmm
Error in format_error_bullets(x[-1]) : 
  nms %in% c("i", "x", "") are not all TRUE
Calls: read_eps_interpolate ... collapse_cnd_message -> paste -> format_error_bullets -> stopifnot
Execution halted

The vfld file is perfectly valid and was handled properly by old Harp. This is used for e.g. IFSENS data where extracting pressure levels from historical data in the MARS archive is still unmanageable.

[uandrae]

Example data can be found under ecgb:/scratch/ms/spsehlam/hlam/test_harp/IFSENS

2020100500 with TEMP and 2020093000 without TEMP.

[andrew-MET]

Could you upload the files with the issue for completeness?

[uandrae]

test_harp.tar.gz

[uandrae]

Any fix in sight?

[andrew-MET]

You can try installing from andrew-MET/harpIO - there is a fix there, but it's not fully tested so I'm not 100% sure it doesn't break other things.

[uandrae]

OK, then I wait for the tested version.

[andrew-MET]

Note that read_eps_interpolate() is now superseded by read_forecast() - development on read_*_interpolate() is now stopped, so I recommend using read_forecast() instead - it is marginally slower, but much more robust and flexible. There are pathways towards speeding it up, but that probably won't be tackled until next year. For reading vfld files the syntax would be:

read_forecast(
  start_date       = ...,
  end_date         = ...,
  fcst_model       = "...",
  parameter        = NULL,
  lead_time        = ...,
  members          = ...,
  file_path        = "...",
  file_template    = "vfld_eps" ,
  output_file_opts = sqlite_opts(path = "...")
)

i.e. it is almost exactly the same except for members instead of members_in and specifying the output file path using output_file_opts = sqlite_opts(path = "..."). You don't need to specify the input file format anymore as it's clever enough to be able to work it out.

See also: https://harphub.github.io/harpIO/articles/read_raw_forecast.html

The reason for this change is that read_forecast() allows us to have a single function for reading both ensemble and deterministic forecasts and doing any transformation on the data such as interpolating to points, regridding / reprojecting and extracting vertical cross sections. Furthermore, it allows us to be able to have different output formats such as csv files, duckDB files, writing to databases of different flavours, etc. etc. should we want to do that in the future.

[uandrae]

Thanks for the guidance. Can confirm that it works now.