search

harphub / harpIO

IO functions for HARP
https://harphub.github.io/harpIO/
Other
6 stars 16 forks source link

Column `p` not found in `.data` #13

Closed mpartio closed 4 years ago

mpartio commented 4 years ago

I'm trying to run vfld to sqlite generation for our new production CMEPS system, but I get the following error:

Error: Column `p` not found in `.data`
<error/rlang_error_data_pronoun_not_found>
Column `p` not found in `.data`
Backtrace:
     █
  1. ├─harpIO::read_eps_interpolate(...)
  2. │ └─`%>%`(...)
  3. │   ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  4. │   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  5. │     └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  6. │       └─harpIO:::`_fseq`(`_lhs`)
  7. │         └─magrittr::freduce(value, `_function_list`)
  8. │           ├─base::withVisible(function_list[[k]](value))
  9. │           └─function_list[[k]](value)
 10. │             ├─dplyr::mutate(...)
 11. │             └─dplyr:::mutate.tbl_df(...)
 12. │               └─dplyr:::mutate_impl(.data, dots, caller_env())
 13. ├─purrr::map2(...

Any help is appreciated!

Harp version is picked from ~mid January. Script:

library(harpIO)

read_eps_interpolate(
 start_date      = Sys.getenv("HARP_BEGINDATE"),
 end_date        = Sys.getenv("HARP_ENDDATE"),
 eps_model       = Sys.getenv("HARP_MODEL"),
 parameter       = NULL,
 by              = Sys.getenv("HARP_FCST_FREQUENCY"),
 lead_time       = seq(0, 54, 3),
 members_in      = eval(parse(text=Sys.getenv("HARP_MEMBERS"))),
 lags            = eval(parse(text=Sys.getenv("HARP_LAGS"))),
 file_path       = Sys.getenv("HARP_VFLD_DIR"),
 file_template   = "ecfadm_vfld{eps_model}/{YYYY}/{MM}/{DD}/vfld{eps_model}mbr{MBR3}{YYYY}{MM}{DD}{HH}{LDT2}",
 sqlite_path     = Sys.getenv("HARP_FCST_SQLITE_DIR"),
 sqlite_template = "{eps_model}/{YYYY}/{MM}/FCTABLE_{parameter}_{YYYY}{MM}_{HH}.sqlite"
)

Environment:

declare -x HARP_BEGINDATE="202002040000"
declare -x HARP_ENDDATE="202002042100"
declare -x HARP_FCST_FREQUENCY="3h"
declare -x HARP_FCST_SQLITE_DIR="/nobackup/opdata/verif/HARPresults/FCtables"
declare -x HARP_LAGS="paste0(c(0,0,0,1,1,2,2,1,2,0,1,2,0,1,2),\"h\")"
declare -x HARP_MEMBERS="0:14"
declare -x HARP_MODEL="MEPS_prod"
declare -x HARP_VFLD_DIR="/data/24_nb/metcoop/opdata/elin4/verif/"

Output:

Warning: No stations specified. Default station list used.
Generating file names.
Reading data for 202002040000
Date assumed to be YYYYMMDDHHmm
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040000
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040000
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040000
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040000
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040000

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040003
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040003
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040003
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040003
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040003

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040006
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040006
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040006
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040006
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040006

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040009
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040009
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040009
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040009
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040009

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040012
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040012
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040012
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040012
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040012

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040015
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040015
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040015
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040015
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040015

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040018
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040018
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040018
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040018
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040018

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040021
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040021
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040021
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040021
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040021

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040024
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040024
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040024
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040024
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040024

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040027
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040027
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040027
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040027
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040027

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040030
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040030
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040030
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040030
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040030

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040033
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040033
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040033
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040033
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040033

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040036
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040036
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040036
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040036
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040036

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040039
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040039
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040039
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040039
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040039

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040042
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040042
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040042
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040042
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040042

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040045
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040045
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040045
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040045
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040045

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040048
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040048
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040048
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040048
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040048

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040051
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040051
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040051
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr009202002040051
Warning: File not found: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr012202002040051

Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040054
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040054
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040054
Warning: Unable to read: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040054

Error: Column `p` not found in `.data`
<error/rlang_error_data_pronoun_not_found>
Column `p` not found in `.data`
Backtrace:
     █
  1. ├─harpIO::read_eps_interpolate(...)
  2. │ └─`%>%`(...)
  3. │   ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  4. │   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  5. │     └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  6. │       └─harpIO:::`_fseq`(`_lhs`)
  7. │         └─magrittr::freduce(value, `_function_list`)
  8. │           ├─base::withVisible(function_list[[k]](value))
  9. │           └─function_list[[k]](value)
 10. │             ├─dplyr::mutate(...)
 11. │             └─dplyr:::mutate.tbl_df(...)
 12. │               └─dplyr:::mutate_impl(.data, dots, caller_env())
 13. ├─purrr::map2(...
In addition: Warning message:
In get("station_list") :
  input string 'Végeirsstadir-i-Fnjoskadal' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?
Execution halted
andrew-MET commented 4 years ago

Can you send me the offending data? If you can find a single lead time where the problem occurs then that would minimise the volume.

mpartio commented 4 years ago

andrew.zip

mpartio commented 4 years ago

It was the last ie 54th hour. Script:

library(harpIO)

read_eps_interpolate(
 start_date      = Sys.getenv("HARP_BEGINDATE"),
 end_date        = Sys.getenv("HARP_ENDDATE"),
 eps_model       = Sys.getenv("HARP_MODEL"),
 parameter       = NULL,
 by              = Sys.getenv("HARP_FCST_FREQUENCY"),
 lead_time       = seq(54, 54, 3),
 members_in      = eval(parse(text=Sys.getenv("HARP_MEMBERS"))),
 lags            = eval(parse(text=Sys.getenv("HARP_LAGS"))),
 file_path       = Sys.getenv("HARP_VFLD_DIR"),
 file_template   = "ecfadm_vfld{eps_model}/{YYYY}/{MM}/{DD}/vfld{eps_model}mbr{MBR3}{YYYY}{MM}{DD}{HH}{LDT2}",
 sqlite_path     = Sys.getenv("HARP_FCST_SQLITE_DIR"),
 sqlite_template = "{eps_model}/{YYYY}/{MM}/FCTABLE_{parameter}_{YYYY}{MM}_{HH}.sqlite"
)

Environment is the same as above. Output:

$ Rscript vfld2sqlite.R 
Warning: No stations specified. Default station list used.
Generating file names.
Reading data for 202002040000
Date assumed to be YYYYMMDDHHmm
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr000202002040054
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr001202002040054
Reading: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040054
Warning: Unable to read: /data/24_nb/metcoop/opdata/elin4/verif//ecfadm_vfldMEPS_prod/2020/02/04/vfldMEPS_prodmbr002202002040054

Error: Column `p` not found in `.data`
<error/rlang_error_data_pronoun_not_found>
Column `p` not found in `.data`
Backtrace:
     █
  1. ├─harpIO::read_eps_interpolate(...)
  2. │ └─`%>%`(...)
  3. │   ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  4. │   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  5. │     └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  6. │       └─harpIO:::`_fseq`(`_lhs`)
  7. │         └─magrittr::freduce(value, `_function_list`)
  8. │           ├─base::withVisible(function_list[[k]](value))
  9. │           └─function_list[[k]](value)
 10. │             ├─dplyr::mutate(...)
 11. │             └─dplyr:::mutate.tbl_df(...)
 12. │               └─dplyr:::mutate_impl(.data, dots, caller_env())
 13. ├─purrr::map2(...
In addition: Warning message:
In get("station_list") :
  input string 'Végeirsstadir-i-Fnjoskadal' cannot be translated to UTF-8, is it valid in 'ANSI_X3.4-1968'?
Execution halted
andrew-MET commented 4 years ago

fixed: 2954e0e9c43bf05e09b26728c69ab5ba0d9e7368

mpartio commented 4 years ago

Works with the latest change. Thanks!