search

harphub / harpIO

IO functions for HARP
https://harphub.github.io/harpIO/
Other
6 stars 16 forks source link

Problem creating obs sqlite with harpIO v0.0.0.9164 #34

Closed uandrae closed 3 years ago

uandrae commented 3 years ago

The following run on ecgate stops after 3days. The sqlite is readable, but incomplete. 

snh@ecgb11:~/harpv3> cat vobs2sqlite.R 
library(harpIO)

first_obs      <- 2019060100
last_obs       <- 2019060921
obs_freq       <- "3h"
#vobs_path       <- "/scratch/ms/spsehlam/hlam/vobs/"
vobs_path       <- "/scratch/ms/se/snh/vobs/prod_SPP_defpert_43h2_1/"
sqlite_path     <- "/scratch/ms/se/snh/harp/OBSTABLE"
#sqlite_path     <- "/scratch/ms/se/snh/harp/OBSTABLE_METCOOP"

read_obs_convert(
  start_date  = first_obs,
  end_date    = last_obs,
  obs_path    = vobs_path,
  sqlite_path = sqlite_path
)

Generates

snh@ecgb11:~/harpv3> cat slurm-36058829.out
Wed Aug 26 12:23:49 GMT 2020
load proj4 5.2.0 (PATH, MANPATH, PROJ4_DIR, EC_FFLAGS, EC_CFLAGS, EC_CXXFLAGS, EC_FLDFLAGS,EC_CLDFLAGS, EC_CXXLDFLAGS)
load R 3.6.3 (PATH)
Reading: /scratch/ms/se/snh/vobs/prod_SPP_defpert_43h2_1//vobs2019060100
Reading: /scratch/ms/se/snh/vobs/prod_SPP_defpert_43h2_1//vobs2019060103
..
Reading: /scratch/ms/se/snh/vobs/prod_SPP_defpert_43h2_1//vobs2019060318

***
New SQLITE obs file created: /scratch/ms/se/snh/harp/OBSTABLE/OBSTABLE_2019.sqlite
***

Writing to: SYNOP in /scratch/ms/se/snh/harp/OBSTABLE/OBSTABLE_2019.sqlite

Writing to: TEMP in /scratch/ms/se/snh/harp/OBSTABLE/OBSTABLE_2019.sqlite

[1] "FAILURE dbSendStatement: 1 / 20"
[1] "no such column: NA"
...
[1] "FAILURE dbSendStatement: 20 / 20"
[1] "no such column: NA"
Error: Unable to to write to database file.
Execution halted
Wed Aug 26 12:25:40 GMT 2020
andrew-MET commented 3 years ago

Can you narrow it down to the vobs files that lead to the problem and attach them to this issue?

uandrae commented 3 years ago

vobs2019060100.gz

uandrae commented 3 years ago

It can be reproduced by the added file

Wed Aug 26 13:02:55 GMT 2020
load proj4 5.2.0 (PATH, MANPATH, PROJ4_DIR, EC_FFLAGS, EC_CFLAGS, EC_CXXFLAGS, EC_FLDFLAGS,EC_CLDFLAGS, EC_CXXLDFLAGS)
load R 3.6.3 (PATH)
Reading: /scratch/ms/se/snh/vobs/prod_SPP_defpert_43h2_1//vobs2019060100

***
New SQLITE obs file created: /scratch/ms/se/snh/harp/OBSTABLE/OBSTABLE_2019.sqlite
***

Writing to: SYNOP in /scratch/ms/se/snh/harp/OBSTABLE/OBSTABLE_2019.sqlite

Writing to: TEMP in /scratch/ms/se/snh/harp/OBSTABLE/OBSTABLE_2019.sqlite

[1] "FAILURE dbSendStatement: 1 / 20"
[1] "no such column: NA"
[1] "FAILURE dbSendStatement: 2 / 20"
[1] "no such column: NA"
[1] "FAILURE dbSendStatement: 3 / 20"
andrew-MET commented 3 years ago

Fixed by #35

Problem was caused by pressure levels set to -99 in the TEMP section of the vobs file. Apparently it hasn't come across that issue before!

uandrae commented 3 years ago

Ah, interesting. It must be a feature of my new EcCodes based BUFR reading. I assume it's not to tricky to fix. In any case I have to check the behavior compared to the old code on my side.