Warnings about httpuv/Rcpp

[trestletech]

I've noticed that httpuv consistently doesn't make it into the Packrat cache for us. It seems like every package I reference gets symlinked in from the cache but httpuv always has to be rebuilt. I'm not sure if that's affiliated with these warnings or not, but I did notice these today and can reproduce them when deploying to the dogfood server.

> deployApp("reactnb/", server="dogfood")
Preparing to deploy application...DONE
Uploading application bundle...DONE
Deploying application: 5...
Deploying to R version '3.1.1' (Local version is '3.0.2')
Loading required package: packrat
Installing RJSONIO (1.0-3) ... OK (symlinked cache)
Installing Rcpp (0.11.1) ... OK (symlinked cache)
Installing bitops (1.0-6) ... OK (symlinked cache)
Installing digest (0.6.4) ... OK (symlinked cache)
Installing packrat (0.4.3) ... OK (symlinked cache)
Installing xtable (1.7-1) ... OK (symlinked cache)
Installing httpuv (1.3.2) ... OK (built source)
Installing caTools (1.14) ... OK (symlinked cache)
Installing htmltools (0.2.6) ... OK (symlinked cache)
Installing shiny (0.10.1) ... OK (symlinked cache)
Warning messages:
1: In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.0.2
2: The following packages specified in the LinkingTo field for package 'httpuv' are unavailable:
- 'Rcpp'
These packages are required to be installed when attempting to hash this package for caching. 
3: The following packages specified in the LinkingTo field for package 'httpuv' are unavailable:
- 'Rcpp'
These packages are required to be installed when attempting to hash this package for caching. 
Application successfully deployed to http://rsc.radixu.com/jeff/reactnb

The first warning is understandable, but the second two are concerning. It looks like Rcpp is not installed in a global library on the dogfood server, but when I deploy to my own test server (which does happen to have Rcpp installed globally) I don't see those warnings, but I still have to rebuild httpuv each time I deploy -- it doesn't get symlinked. e.g.

> deployApp("reactnb/", server="dev")
Preparing to deploy application...DONE
Uploading application bundle...DONE
Deploying application: 11...
Deploying to R version '3.1.2' (Local version is '3.0.2')
Loading required package: packrat
Installing RJSONIO (1.0-3) ... OK (symlinked cache)
Installing Rcpp (0.11.1) ... OK (symlinked cache)
Installing bitops (1.0-6) ... OK (symlinked cache)
Installing digest (0.6.4) ... OK (symlinked cache)
Installing packrat (0.4.3) ... OK (symlinked cache)
Installing xtable (1.7-1) ... OK (symlinked cache)
Installing httpuv (1.3.2) ... OK (built source)
Installing caTools (1.14) ... OK (symlinked cache)
Installing htmltools (0.2.6) ... OK (symlinked cache)
Installing shiny (0.10.1) ... OK (symlinked cache)
Warning message:
In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.0.2
Application successfully deployed to http://10.0.0.10:3939/admin/reactnb
> 

If you'd prefer, I can open a separate issue for the httpuv caching issue if that looks unrelated to you.

[kevinushey]

Debugging this seems like it will be fun... it seems like there a number of interacting issues here.

First, let's recall that we attempt to hash LinkingTo dependencies recursively, to avoid SNAFUs with e.g. pkgA linked with pkgB 1.0 vs. pkgB 1.1.

When checking whether a package exists in the cache or not, we should be comparing the hashes contained within the packrat lockfile, and the directory names within the cache directory. If we find a hit, we use that. For whatever reason, the hashes don't seem to match for the packrat lockfile and the cache, for httpuv. (Can you confirm if that's the case?)

If, for some reason, we cannot use the cache, we attempt to install from source, and then re-populate the cache after installing from source. All of the pertinent code for hashing is here: https://github.com/rstudio/packrat/blob/master/R/cache.R. In particular, we attempt to locate the DESCRIPTION file for Rcpp with:

system.file("DESCRIPTION", package = "Rcpp")

and it appears that this is failing (Rcpp is not found). I'm guessing that this is failing because the package exists in the cache, but not in the local library, and so system.file() doesn't know where to look up Rcpp? And because of that, httpuv never enters the cache -- does that seem like a possibility?

For the second case, where you don't see any warnings, I would expect that after the restore, you should now see httpuv has entered the cache, and the hash for httpuv and the cache entry now match. I am guessing that this is not the case, for some reason, if the cache never gets used when attempting to restore the project.

[trestletech]

Yeah, it looks like httpuv has different hashes. In the packrat cache, we have 5d0ed1b9f94a37f8947909a7f4aeafd1

and the packrat lock file has

Package: httpuv
Source: CRAN
Version: 1.3.2
Hash: 74aa6ae7984396db996ca6d91167177c
Requires: Rcpp

I'm guessing that this is failing because the package exists in the cache, but not in the local library

That's certainly possible. At this point in the code, we load Packrat from a library that we manage ourselves (to ensure that Packrat exists, whether or not it had previously been installed on the system), so unless Rcpp is in a system library (which it's not on this machine), Rcpp would definitely not be found on the system. Should packrat import Rcpp? Or should we consider that internally as a dependency that won't be fulfilled by install.packages (since perhaps it's unlikely that people will be using the cache)? We can easily install Rcpp in our own library the same way we do for Packrat.

In the second case, I still only see one version of httpuv in the cache, despite having seen it build multiple times.

Let me know what other info you need or I can get you access to either/both of these machines sometime.