wlandau
commented
6 years ago @noamross this is such an important point. As you say, drake may not be the place to build in this functionality, but I think it should at least have more guidance. The best practices guide and the cranlogs example should talk about remote data, and there could even be a whole separate vignette on remote data and big data.
I think I have a solution you can use right now. Does the code below cover it? The general principle applies to all remote data, and the download trigger could vary: you could pick a DOI, a time stamp, a hash, or something else. (Related: see triggers in workflow plans.) Also, how do you get a hash from HEAD()? Below, I use the time stamp, but the hash would be much better.
# Goal: get some cran download logs
# and only do a download when the remote time stamp changes.
library(drake)
library(curl)
library(httr)
library(R.utils)
url <- "http://cran-logs.rstudio.com/2018/2018-02-09-r.csv.gz"
query <- HEAD(url)
# The timestamp is an imported object.
# I use it as a dependency of the target.
# Every time you run this R script, we get a new timestamp.
# We will download from the url if and only if:
# - The timestamp changed.
# - The URL changed.
# - The code changed nontrivially.
# - We don't already have the target.
timestamp <- query$headers[["last-modified"]]
timestamp
## [1] "Mon, 12 Feb 2018 16:34:48 GMT"
# The ... is for other dependencies that should trigger new downloads.
get_logs <- function(url, ...){
curl_download(url, "logs.csv.gz") # Get a big file.
gunzip("logs.csv.gz", overwrite = TRUE) # Unzip it.
out <- read.csv("logs.csv", nrows = 4) # Extract the data you need.
unlink(c("logs.csv.gz", "logs.csv")) # Remove the big files
out # Value of the target.
}
cranlogs_plan <- drake_plan(
# The timestamp is a dependency, so it triggers a new download.
# You could just put the body of get_logs() here,
# but it makes it harder to read.
logs = get_logs(url, timestamp)
)
# Let's see what the graph looks like.
config <- drake_config(cranlogs_plan)
vis_drake_graph(config)
make(cranlogs_plan)
## ...
## target logs
readd(logs)
## date time size version os country ip_id
## 1 2018-02-09 13:01:13 82375220 3.4.3 win RO 1
## 2 2018-02-09 13:02:06 74286541 3.3.3 win US 2
## 3 2018-02-09 13:02:10 82375216 3.4.3 win US 3
## 4 2018-02-09 13:03:30 82375220 3.4.3 win IS 4
# The logs are up to date now.
vis_drake_graph(config)
# But what if our timestamp changes?
timestamp <- "new day"
outdated(config)
## [1] "logs"# Fun fact: you could also use the timestamp
# as a dependency of get_logs() and `logs` will
# react the same way.
# The ... is for other dependencies that should trigger new downloads.
get_logs <- function(url){
timestamp # Now a dependency of get_logs()
curl_download(url, "logs.csv.gz") # Get a big file.
gunzip("logs.csv.gz", overwrite = TRUE) # Unzip it.
out <- read.csv("logs.csv", nrows = 4) # Extract the data you need.
unlink(c("logs.csv.gz", "logs.csv")) # Remove the big files
out # Value of the target.
}
cranlogs_plan <- drake_plan(
# The timestamp is a dependency, so it triggers a new download.
# You could just put the body of get_logs() here,
# but it makes it harder to read.
logs = get_logs(url)
)
config <- drake_config(cranlogs_plan)
vis_drake_graph(config) # The `timestamp` import points to `get_logs` now.
outdated(config)
## [1] "logs"
make(cranlogs_plan)
## target logs
outdated(config)
## character(0)
make(cranlogs_plan)
## All targets are already up to date.
# Drake knows that the new timestamp affects get_logs,
# which puts all downstream targets into question.
timestamp <- "yet another day"
outdated(config)
## [1] "logs"
make(cranlogs_plan)
## target logs
noamross
willpearse
sckott
tiernanmartin
ldecicco-USGS
A frequent challenge to both efficient, reproducible, and compact workflows is acquiring data from external sources. For instance, on a current project, I download Facebook's pre-trained fasttext word embedding model which is ~6GB. This takes a while. I don't want to include this file in the repository, instead putting code in the repo so collaborators get it as part of the workflow. I may or may want to update this when the remote sources changes. If I do, unfortunately, I need to download the whole thing to hash the file and check if it is different than before.
However, for many cases, there are strong heuristics that would allow me to check if the remote file is updated prior deciding whether to download. I think drake, or a small helper package that drake would use, seems a great home for functions to use these heuristics.
Some heuristics may be available in a file's content header which can be queried via a
HEAD()call without downloading a file. These include:If the file is on an FTP server, the modified date and size should be available as well.
Furthermore, based on parsing the URL itself, one could ascertain information on the file state for many important sources.:
So, if drake detects a URL in a work plan, like this
It could run the url against something like
spyglass::check_remote_file(url, spyobj), wherespyobjholds all the available heuristic checks. Checking against this object would tell you if the file has changed, and also what heuristics it is based on. drake could keepspyobjin the cache for the next lookup. drake users could decide on heuristics they would accept as a work plan option (hash or date-modified + size probably being the default).(spyglass package name comes free.)
OK, maybe this is a bit out of scope and we should build this package separately, but interested in your thoughts!
Tagging @sckott and @willpearse based on their suppdata package and related DOI-parsing packages