search

harphub / harpIO

IO functions for HARP
https://harphub.github.io/harpIO/
Other
6 stars 16 forks source link

Crash in read_eps_interpolate #10

Closed uandrae closed 4 years ago

uandrae commented 4 years ago

The following vfld2sqlite conversion 

[mcoop.u@lxserv1683 testharp]$ cat testme.R 
library(harpIO)

read_eps_interpolate(
 start_date      = "201910010000",
 end_date        = "201910082100",
 eps_model       = "CMEPS_preop",
 parameter       = NULL,
 by              = "3h",
 lead_time       = seq(0,63,3),
 members_in      = 0:8,
 lags            = paste0(c(0,0,0,2,2,1,1,2,1),"h"),
 file_path       = "/data/proj/metcoop/mc-20-utv/opdata/verif",
 file_template   = "ecfadm_vfld{eps_model}/vfld{eps_model}mbr{MBR3}{YYYY}{MM}{DD}{HH}{LDT2}",
 sqlite_path     = ".",
 sqlite_template = "{eps_model}/{YYYY}/{MM}/FCTABLE_{parameter}_{YYYY}{MM}_{HH}.sqlite"
)

fails with the message below. The error cannot be reproduced when starting on e.g. the 6th but I haven't tried all possibilities. The conversion started without any FCtables present. For the full error message check lxserv1683:/local_disk/testharp/logme


Writing to: ./CMEPS_preop/2019/10/FCTABLE_Tmin_201910_18.sqlite

Writing to: ./CMEPS_preop/2019/10/FCTABLE_Gmax_201910_18.sqlite

Writing to: ./CMEPS_preop/2019/10/FCTABLE_Z_201910_18.sqlite

<Rcpp::exception: UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID>
[1] "FAILURE dbSendQuery 1 / 20"
[1] "UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID"
<Rcpp::exception: UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID>
[1] "FAILURE dbSendQuery 2 / 20"
[1] "UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID"
<Rcpp::exception: UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID>
[1] "FAILURE dbSendQuery 3 / 20"
[1] "UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID"
<Rcpp::exception: UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID>
[1] "FAILURE dbSendQuery 4 / 20"
[1] "UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID"
<Rcpp::exception: UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID>
[1] "FAILURE dbSendQuery 5 / 20"
[1] "UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID"
<Rcpp::exception: UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID>
...
[1] "UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID"
<Rcpp::exception: UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID>
[1] "FAILURE dbSendQuery 20 / 20"
[1] "UNIQUE constraint failed: FC.fcdate, FC.leadtime, FC.SID"
Error in dbquery(db_conn, SQL_add_index) : 
  FATAL DBFAILURE: Unable to query database.
Calls: read_eps_interpolate ... <Anonymous> -> map2 -> .f -> db_clean_and_write -> dbquery
In addition: Warning message:
In get("station_list") :
  input string 'Végeirsstadir-i-Fnjoskadal' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?
Execution halted
andrew-MET commented 4 years ago

The problem was caused when there are no data for a lagged member - the initial check is done on all members for a specific time i.e. lagged and unlagged together, but they are split into original forecast times for writing so if there are no data for a lagged time into tried to still write the data. A final check is now done before writing: https://github.com/andrew-MET/harpIO/commit/33a466b8c8b1e86073cc791722e2e1d5ff3cc2b9