download_historic_data has parsing failure

[joethorley]

@ateucher It seems download_historic_data has a parsing failure - I may have spoken prematurely when I said it was fixed...sorry to be the bearer of bad news :(

R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin13.4.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> devtools::install_github("bcgov/rems")
Using GitHub PAT from envvar GITHUB_PAT
Skipping install of 'rems' from a github remote, the SHA1 (5c938fdd) has not changed since last install.
  Use `force = TRUE` to force installation
> library(rems)
> download_historic_data(force = TRUE, ask = FALSE)
This is going to take a while...
Downloading latest 'historic' EMS data from BC Data Catalogue (url:https://pub.data.gov.bc.ca/datasets/949f2233-9612-4b06-92a9-903e817da659/ems_sample_results_historic_expanded.zip)
  |=============================================================================================| 100%
Saving historic data at /Users/joe/Library/Application Support/rems/ems.sqlite
|=
Warning: 4956875 parsing failures.
row                    col   expected                             actual
  1 RESULT                 a double   Iron Dissolved                    
  1 METHOD_DETECTION_LIMIT a double   Filt;HNO3 dig;conc'n;nebul'ICP-OES
  1 LOWER_DEPTH            a double   mg/L                              
  1 AIR_FILTER_SIZE        a double   C                                 
  1 NA                     55 columns 59 columns                        
... ...................... .......... ..................................
See problems(...) for more details.

Error: 4956875 parsing failures
> traceback()
5: stop(n, " parsing failure", if (n > 1) "s", call. = FALSE)
4: readr::stop_for_problems(ret)
3: read_ems_data(csv_file, n = n, cols = col_names, verbose = FALSE, 
       skip = skip, col_names = col_names, progress = FALSE)
2: save_historic_data(csv_file, db_path, n)
1: download_historic_data(force = TRUE, ask = FALSE)
> 

[ateucher]

They added a few new columns to the csvs on the server. I'll add them.

[joethorley]

Thanks

[HeatherGranger]

Thanks Andy! We're these columns added to the historical data? I can't open the spreadsheet to check...

[ateucher]

Yup, and to the current as well. I'm just doing a bit of cleanup - I'll let you know when it's ready to test!

[joethorley]

Fixed at my end

[ateucher]

@HeatherGranger It should be good to go with the new columns added. You'll need to reinstall from github, and probably want to refresh your cache:

devtools::install_github("bcgov/rems")
library(rems)
remove_data_cache("all")

Then you can use get_ems_data() or download_ems_data(); read_historic_data() to import the data with the new columns.