read_eps_interpolate doesn't handle missing data properly

[uandrae]

The following test

library(harpIO)

read_eps_interpolate(
 start_date      = "201908260900",
 end_date        = "201908260900",
 eps_model       = "CMEPS_preop",
 parameter       = "T2m",
 by              = "3h",
 lead_time       = seq(57,63,3),
 members_in      = 3,
 lags            = paste0(c(2),"h"),
 file_path       = "/data/proj/metcoop/mc-20-utv/opdata/verif/",
 file_template   = "ecfadm_vfld{eps_model}/vfld{eps_model}mbr{MBR3}{YYYY}{MM}{DD}{HH}{LDT2}",
 sqlite_path     = "/nobackup/opdata/verif/HARPresults/FCtables",
 sqlite_template = "{eps_model}/{YYYY}/{MM}/FCTABLE_{parameter}_{YYYY}{MM}_{HH}.sqlite"
)

Generates and error as shown below. We must be able to handle missing data properly.

[mcoop.u@lxserv1683 ~]$ Rscript /local_disk/a000864_build_area/metcoop-harp/metcoop-harp/Harp/eps/R/testme.R Warning: No stations specified. Default station list used. Generating file names. Reading data for 201908260900 Date assumed to be YYYYMMDDHHmm Warning: File not found: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr003201908260759

Warning: File not found: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr003201908260762

Warning: File not found: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr003201908260765

Joining data. Preparing data to write. Error in format.POSIXlt(mt, "%d", usetz = FALSE) : zero-length component [[5]] in non-empty "POSIXlt" structure Calls: read_eps_interpolate ... + -> add_period_to_date -> add_months -> format.POSIXlt In addition: Warning message: In get("station_list") : input string 'Végeirsstadir-i-Fnjoskadal' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'? Execution halted

[uandrae]

Adding data for an existing member, with a different lagging, overcomes this problem. However for the parameter D (TEMP wind direction ) we get a failure?

read_eps_interpolate(
 start_date      = "201908260900",
 end_date        = "201908260900",
 eps_model       = "CMEPS_preop",
 parameter       = "D",
 by              = "3h",
 lead_time       = seq(57,63,3),
 members_in      = seq(2,3,1),
 lags            = paste0(c(0,2),"h"),
 file_path       = "/data/proj/metcoop/mc-20-utv/opdata/verif/",
 file_template   = "ecfadm_vfld{eps_model}/vfld{eps_model}mbr{MBR3}{YYYY}{MM}{DD}{HH}{LDT2}",
 sqlite_path     = "/nobackup/opdata/verif/HARPresults/FCtables",
 sqlite_template = "{eps_model}/{YYYY}/{MM}/FCTABLE_{parameter}_{YYYY}{MM}_{HH}.sqlite"
)

generates

[mcoop.u@lxserv1683 ~]$ Rscript  /local_disk/a000864_build_area/metcoop-harp/metcoop-harp/Harp/eps/R/testme.R 
Warning: No stations specified. Default station list used.
Generating file names.
Reading data for 201908260900
Date assumed to be YYYYMMDDHHmm
Reading: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr002201908260957
Reading: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr002201908260960
Reading: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr002201908260963
Warning: File not found: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr003201908260759

Warning: File not found: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr003201908260762

Warning: File not found: /data/proj/metcoop/mc-20-utv/opdata/verif//ecfadm_vfldCMEPS_preop/vfldCMEPS_preopmbr003201908260765

Joining data.
Preparing data to write.
Writing to: /nobackup/opdata/verif/HARPresults/FCtables/CMEPS_preop/2019/08/FCTABLE_D_201908_09.sqlite

Writing to: /nobackup/opdata/verif/HARPresults/FCtables/CMEPS_preop/2019/08/FCTABLE_D_201908_07.sqlite

<Rcpp::exception: UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID>
[1] "FAILURE dbSendQuery 1 / 20"
[1] "UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID"
^C

[andrew-MET]

We must be able to handle missing data properly.

Please just state the issue rather than critical remarks. Please bear in mind this is still in development and you are in effect beta testing so should expect things not to work in every situation.

Missing data were handled fine until I added the ability to still create the correct tables when individual members are missing at the start. To do this means carrying around some fake data and dropping them just before the INSERT. Unfortunately after making the change I didn't think to test the situation where there are no data at all. This is now fixed in https://github.com/andrew-MET/harpIO/commit/17e9c2ca8ab805faeff2f0e3bf5b3b7546940c01. The second problem I cannot replicate whatever I try. If you delete the offending files first, is it still a problem?

[uandrae]

I'm aware that it's work in progress. Wi'll try with the latest updates and report back the findings.