search

bcgov / rems

An R package to access data from British Columbia's Environmental Monitoring System
Apache License 2.0
19 stars 5 forks source link

Sqlite direct download #37

Closed ateucher closed 5 years ago

ateucher commented 5 years ago

This PR does two major things:

  1. Provides the historic sqlite database as a direct download from a rems release. This means that download_historic_data() does not involve reading the csv from the BCDC and iteratively loading it into sqlite. That process is now done by the package maintainer and download_historic_data() simply downloads the sqlite file and stores it in the rems data dir.

  2. Allow a user to specify/customize the location of the sqlite database via an option rems.historic.path - (getOption("rems.historic.path", default = rems_data_dir()))

ateucher commented 5 years ago

To test:

remotes::install_github("bcgov/rems@sqlite-path")
rems:::burn_it_down()

### Restart R session before proceeding ###

library(rems)
download_historic_data()
read_historic_data(emsid = "0400203", from_date = as.Date("1984-11-20"),
                   to_date = as.Date("1991-05-11"))
ateucher commented 5 years ago

@HeatherGranger @sebdalgarno

sebdalgarno commented 5 years ago

this is a great idea @ateucher ! I have followed the instructions above. I am able to initiate downloading the data but have now failed on 5 attempts (all at different stages of completion). I do feel that my poor Haida Gwaii may be the culprit. Hopefully someone else can test as well to confirm.

sebdalgarno commented 5 years ago

@HeatherGranger can you try restarting your R session prior to running library(rems) and all that follows?

ateucher commented 5 years ago

Ah yes, I forgot to mention that. Instructions updated

HeatherGranger commented 5 years ago

That worked. It's currently downloading. I'll report back on it's success!

HeatherGranger commented 5 years ago

Success! downloaded, stored and read in the data.

ateucher commented 5 years ago

Thanks for testing @HeatherGranger - that's great!

ateucher commented 5 years ago

@sebdalgarno I've just pushed a commit to allow passing httr::GET configurations. Can you try with a really long timeout threshold (timeout() value is in seconds)? eg. 

download_historic_data(ask = FALSE, force = TRUE, httr_config = timeout(5000))
sebdalgarno commented 5 years ago

@ateucher I'm trying now with timeout(5000) as suggested. Will report back soon!

boshek commented 5 years ago

Similar success over here 🎉

ateucher commented 5 years ago

Thanks @boshek - that's great!

sebdalgarno commented 5 years ago

Well I set up a better cell signal booster in my tree this morning and can now say I have successfuly downloaded the historic dataset! thanks @ateucher for the changes.

ateucher commented 5 years ago

Excellent! I have a bit more cleanup to do then I will merge this to master

ateucher commented 5 years ago

One thing I haven't tested is updating if you had an old version without using burn_it_down(). @J-krogh can you please try the above without using burn_it_down()? i.e.:

remotes::install_github("bcgov/rems@sqlite-path")
library(rems)
download_historic_data()
read_historic_data(emsid = "0400203", from_date = as.Date("1984-11-20"),
                   to_date = as.Date("1991-05-11"))
j-krogh commented 5 years ago

@ateucher done, all seems to be working!