Packages depending on other packages fail with packrat 0.5.0

karltk commented 5 years ago

Problem

I am trying to install the R package pillar from CRAN using Packrat 0.5.0 on R 5.2.1, but for some reason the installation process breaks because the prerequisite package crayon is not detected even though it was automatically installed by install.packages.

Steps to reproduce

Create a dummy directory for a minimal project:

$ mkdir ~/zzz
$ echo 'packrat::init("~/zzz")' | R --no-save

Run the R interpreter inside this directory:
```
$ cd ~/zzz
$ R
```
On startup, R acknowledges the Packrat installation:
```
Packrat mode on. Using library in directory:
- "~/zzz/packrat/lib"
```

Try to install the pillar package:

> install.package("pillar")
Installing package into ‘/home/karltk/zzz/packrat/lib/x86_64-pc-linux-gnu/3.5.2’
(as ‘lib’ is unspecified)
also installing the dependencies ‘assertthat’, ‘cli’, ‘crayon’, ‘fansi’, ‘rlang’, ‘utf8’
... [SNIP]
* installing *source* package ‘crayon’ ...
** package ‘crayon’ successfully unpacked and MD5 sums checked
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (crayon)
... [SNIP]
* installing *source* package ‘pillar’ ...
... [SNIP]
** testing if installed package can be loaded
Error: package or namespace load failed for ‘pillar’:
.onLoad failed in loadNamespace() for 'pillar', details:
call: loadNamespace(name)
error: there is no package called ‘crayon’
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/home/karltk/zzz/packrat/lib/x86_64-pc-linux-gnu/3.5.2/pillar’

Verify that the crayon library actually exists and is accessible:
```
library(crayon)
?crayon
```

Environment information

Output from R sessionInfo:

R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=nb_NO.utf8        LC_NUMERIC=C              
 [3] LC_TIME=nb_NO.utf8         LC_COLLATE=nb_NO.utf8     
 [5] LC_MONETARY=nb_NO.UTF-8    LC_MESSAGES=en_US.utf8    
 [7] LC_PAPER=nb_NO.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=nb_NO.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.5.2

Output from cat /etc/lsb-relase:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

Output from R:

> packrat::status()
Up to date.

Discussion

I've tried with install.packages('pillar', quiet = FALSE, verbose = TRUE) to dig deeper into the issue, but not found any additional information that I was able to use. It seems such a basic problem that I suspect this might be a PEBCAC-situation, but I've been unable to figure out why the reshape2 package installs just fine, whereas the pillar package does not.

kevinushey commented 5 years ago

That's very odd. I wasn't able to reproduce either on macOS or my Ubuntu 18.04 VM.

My best guess is that perhaps you have an R startup profile somewhere that's trying to set the library paths, and that's somehow conflicting with the set of library paths used during package install in the Packrat project.

karltk commented 5 years ago

That's a very good guess! I do indeed call .libPaths in my ~/.Rprofile, but I thought that was harmless because

The packrat docs state, on the caveats/limitations page page, that "Any local or site-wide initialization file is overridden.", and
The lib paths looked reasonable to me, from inside the project:
```
> .libPaths()
[1] "/home/karltk/zzz/packrat/lib/x86_64-pc-linux-gnu/3.5.2"    
[2] "/home/karltk/zzz/packrat/lib-ext/x86_64-pc-linux-gnu/3.5.2"
[3] "/home/karltk/zzz/packrat/lib-R/x86_64-pc-linux-gnu/3.5.2"  
```
However, it does indeed appear that the settings in ~/.Rprofile are being picked up somehow, and that they're messing with the packrat environment. Removing the call to .libPaths in ~/.Rprofile resolves the issue.

Our of curiousity, would you happen to know if there's a another way to detect this kind of library path confusion from inside packrat-enabled projects? Evidently, inspecting .libPaths() isn't enough, because its output is not affected by what I put in ~/.Rprofile in this case.

kevinushey commented 5 years ago

My understanding is that during install.packages(), R invokes the moral equivalent of:

R CMD INSTALL -l <library> <package>

That process should be launched in the current working directory, and so will use whatever startup profile would normally be activated for that path.

To wit, I would only expect the behavior you're seeing if you were calling install.packages() when not within the project's directory, but I could be wrong.

karltk commented 5 years ago

Hmm. That's what I'm doing. I'm changing directory and starting R inside the packrat-enabled project dir). I've verified that the directory has its own .Rprofile generated by packrat. But it may be that subprocesses invoked by install.packages somehow pick up the info from ~/.Rprofile for some reason.

Anyway, thanks for your kind assistance. Feel free to close this ticket. (You may or may not want to consider adding a small warning / qualification to the caveats section in the docs for others who might fall into the same trap).

karltk commented 5 years ago

For future reference: It appears that a temporary directory is created for each package that is being installed, e.g.

/tmp/RtmpTZXCMH/R.INSTALL498a6ea1eddd/HDF5Array
/tmp/Rtmp67ytex/R.INSTALL49a036c22af8/Rsamtools
/tmp/RtmpJuwIqI/R.INSTALL4a6c6da7f761/doRNG
/tmp/RtmppZ6ONN/R.INSTALL4a7c2976b671/dplyr

and that R is invoked inside of each one of these. As there is no .Rprofile in these directories, R presumably it falls back to sourcing ~/.Rprofile, which in my case overwrote the .libPath.

kevinushey commented 5 years ago

What if you set Sys.setenv(R_PROFILE_USER = "") before running the command -- does that work around the issue?

Brawni commented 4 years ago

Hi Kevin,

Just wanted to say that I had the same issue and solved by adding: Sys.setenv(R_PROFILE_USER = "") as you suggested before installing packages.

Cheers!

rstudio / packrat