packrat::init fails with an Rcpp package installed using devtools::install_github

[jburos]

Hi,

Not sure what to title this issue since I haven't quite isolated the problem. I have, however, been able to reproduce it.

Packrat seems to fail when working with packages I have installed from github & which have compiled sources. These packages (because they are compiled) also tend to be installed with particular args passed to the command to devtools::install_github. I have finally also found that a restart of the R process after installing is usually necessary before loading the package.

One of these steps in the above is likely responsible for the failure to cleanly initialize a project using packrat where just such a package has been included in the snapshot.

I have set up an example repo to illustrate the issue here.

For example, say I have cloned this repo (here I am cloning into another directory, to not conflict with that in which I created the repo):

git clone git@github.com:jburos/test-packrat-restore.git test-packrat-restore-v2
cd test-packrat-restore-v2
R

I would see:

 [ ... ]  ## running `source("packrat/init.R")`

Installing colourpicker (0.3) ...
        OK (built source)
Installing shinystan (2.3.0) ...
        OK (built source)
Installing rstanarm (2.13.1) ...
Error: Command failed (1)

Failed to run system command:

        '/usr/lib/R/bin/R' --vanilla CMD INSTALL '/tmp/Rtmpx1OFWx/stan-dev-rstanarm-db6974c' --library='/home/jacquelineburos/projects/test-packrat-restore-v2/packrat/lib/x86_64-pc-linux-gnu/3.3.2' --install-tests --no-docs --no-multiarch --no-demo

The command failed with output:
* installing *source* package 'rstanarm' ...
** libs
** R
** data
*** moving datasets to lazyload DB
** exec
** inst
** tests
** preparing package for lazy loading
** help
No man pages found in package  'rstanarm'
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error in library.dynam(lib, package, package.lib) :
  shared object 'rstanarm.so' not found
Error: loading failed
Execution halted
ERROR: loading failed
* removing '/home/jacquelineburos/projects/test-packrat-restore-v2/packrat/lib/x86_64-pc-linux-gnu/3.3.2/rstanarm'

Note that the code I used to install rstanarm is as follows (included in the repo as example_script.R):

install.packages('devtools')
library(devtools)
if (!require('rstanarm')) {
  devtools::install_github('stan-dev/rstanarm', ref='feature/jm', args='--preclean', local=FALSE)
  library('rstanarm')
}

After that, I ran packrat::snapshot(), committed changes & pushed to the github repo.

I would also note, upon "trying again" (IE quitting & restarting R in the same session), I see the following:

jacquelineburos@jacki2:~/projects/test-packrat-restore-v2$ R

R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Packrat mode on. Using library in directory:
- "~/projects/test-packrat-restore-v2/packrat/lib"
> library(rstanarm)
Error in library(rstanarm) : there is no package called ‘rstanarm’
> packrat::status()

The following packages are used in your code, tracked by packrat, but no longer present in your library:
                 from   to
    rstanarm   2.13.1   NA

Use packrat::restore() to restore these libraries.

However, running packrat::restore() would yield the same error message as was included above.

In practice, this leads to unexpected & confusing behavior in a real-world scenario since in practice I would normally have several packages that would not have been installed after the error initializing & no way to trigger their reinstall. I am also quite hesitant to direct a colleague to use this repository without explanation.

As it stands, the best work-around I have found to use this type of package with packrat is to include each of these packages in the ignored.packages list & modify the .Rprofile to include a script to install it manually, as done above. Any other are suggestions welcome.

[kevinushey]

Unfortunately, Packrat doesn't currently have a mechanism for passing along install arguments / configuration arguments when installing packages from source.

When I attempt to install rstanarm outside of the context of Packrat, I see the same error:

> devtools::install_github("stan-dev/rstanarm")
Downloading GitHub repo stan-dev/rstanarm@master
from URL https://api.github.com/repos/stan-dev/rstanarm/zipball/master
Installing rstanarm
'/Library/Frameworks/R.framework/Resources/bin/R'  \
  --no-site-file --no-environ --no-save --no-restore --quiet  \
  CMD INSTALL  \
  '/private/var/folders/tm/5dt8p5s50x58br1k6wpqnwx00000gn/T/RtmpImyp0s/devtools81a77296138d/stan-dev-rstanarm-7a68d1e'  \
  --library='/Users/kevin/Library/R/3.3/library'  \
  --with-keep.source --install-tests 

* installing *source* package ‘rstanarm’ ...
** libs
** R
** data
*** moving datasets to lazyload DB
** demo
** exec
** inst
** tests
** preparing package for lazy loading
** help
No man pages found in package  ‘rstanarm’ 
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error in library.dynam(lib, package, package.lib) : 
  shared object ‘rstanarm.so’ not found
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Users/kevin/Library/R/3.3/library/rstanarm’
Error: Command failed (1)

Is there a reason why rstanarm seems to require the use of --preclean?

[jburos]

The reason for --preclean is to remove files from a previous run, particularly those which may have been compiled (by StanHeaders, for example) for another version of the package.

The "shared object 'rstanarm.so' not found" error is, I believe, a separate issue. I have found that the R process needs to restart in order to find this object.

Both of these are noted in my second paragraph above; either is actually sufficient to induce a failure of packrat::init() to work. In your case, I assume that this is your first time installing stan-dev/rstanarm and so I suspect it's the latter scenario.

Curious to know if you have any thoughts on how we might make packrat::init() more reliable in these scenarios?