search

rstudio / packrat

Packrat is a dependency management system for R
http://rstudio.github.io/packrat/
402 stars 90 forks source link

packrat symlink behavior causes errors when running many R sessions simultaneously #276

Open machow opened 8 years ago

machow commented 8 years ago

Sorry, this is my last issue! (and thanks for all the help, packrat is a fantastic tool :)

I'm running simulations in R on a computer cluster, by running the same Rscript job on many nodes in parallel. However, since packrat deletes and recreates symlinks, it causes several errors across jobs (in a sporadic fashion).

I can work around this by staggering each job, so they don't begin at the same time. However, it seems like an option to disable the deletion and recreation of symlinks would bring peace of mind when turning packrat mode on! Going further, an option to disable the creation or deletion of anything, as if running packrat in "read-only" mode seems like it would be valuable for making packrat mode very simple and intuitive.

errors with base package symlinks

Error in if (nt == 1 && file.exists(to) && file.info(to)$isdir) to <- file.path(to,  : 
  missing value where TRUE/FALSE needed
Calls: source ... setPackratModeOn -> afterPackratModeOn -> symlinkSystemPackages
In addition: Warning messages:
1: In readLines(file) :
  cannot open compressed file '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-R/base/DESCRIPTION', probable reason 'No such file or 
directory'
2: In value[[3L]](cond) :
  Unable to read DESCRIPTION file associated with 'base' package
Execution halted

warnings with external package symlinks (I think I've gotten errors here also, but would need to start a bunch of jobs and cross my fingers)

Warning messages:
1: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-ext/RCurl', reason 'File ex
ists'
2: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/bitops' to '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-ext/bitops/bitops', reason
 'File exists'
3: In symlinkExternalPackages(project = project) :
  The following external packages could not be linked into the packrat private library:
- 'RCurl', 'bitops'
4: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-ext/RCurl/RCurl', reason 'F
ile exists'
5: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/bitops' to '/jukebox/hasson/michael/dissertation/paper-pattern-similarity/packrat/lib-ext/bitops/bitops', reason
 'File exists'
6: In symlinkExternalPackages(project = project) :
  The following external packages could not be linked into the packrat private library:
- 'RCurl', 'bitops'

I dug around a bit in the source code to figure out what was going on, and am happy to take a pass at implementing something if pointed in the right direction.

kevinushey commented 8 years ago

I think the problem here is that packrat over-aggressively destroys and re-creates these symlinks every time it's activated for a project. Used here:

https://github.com/rstudio/packrat/blob/222058723d7a8474e670cfc1d0b0b23bd4d2996c/R/packrat-mode.R#L126-L129

https://github.com/rstudio/packrat/blob/222058723d7a8474e670cfc1d0b0b23bd4d2996c/R/packrat.R#L221-L222

And implementation:

https://github.com/rstudio/packrat/blob/222058723d7a8474e670cfc1d0b0b23bd4d2996c/R/library-support.R#L1-L86

It looks like there was some initial provisions added to detect if the libR directory needed to be re-synced, but wasn't followed through all the way.

kevinushey commented 8 years ago

Hi @machow,

This should be resolved with the latest development version of packrat -- can you give it a shot with

devtools::install_github('rstudio/packrat')

and let me know how it goes? (The PR should ensure that symlinks are only re-generated if necessary)

Thanks!

machow commented 8 years ago

I'm still getting both the errors and warnings about external packages. I'll try to look back at where the symlinks are being generated in the updated packrat tomorrow.

sessionInfo() called from within job:

R version 3.1.1 (2014-07-10)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  base     

loaded via a namespace (and not attached):
[1] packrat_0.4.6-20 tools_3.1.1

Error:

Error in if (nt == 1 && file.exists(to) && file.info(to)$isdir) to <- file.path(to,  : 
  missing value where TRUE/FALSE needed
Calls: source ... symlinkExternalPackages -> lapply -> FUN -> symlink -> file.symlink
In addition: Warning message:
In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/test-packrat/packrat/lib-ext/RCurl/RCurl', reason 'File exists'

Warnings:

Warning messages:
1: In dir.create(libExtDir(project), recursive = TRUE) :
  '/jukebox/hasson/michael/dissertation/test-packrat/packrat/lib-ext' already exists
2: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/test-packr
at/packrat/lib-ext/RCurl/RCurl', reason 'File exists'
3: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/bitops' to '/jukebox/hasson/michael/dissertation/test-pack
rat/packrat/lib-ext/bitops/bitops', reason 'File exists'
4: In symlinkExternalPackages(project = project) :
  The following external packages could not be linked into the packrat private library:
- 'RCurl', 'bitops'
Warning messages:
1: In dir.create(libExtDir(project), recursive = TRUE) :
  '/jukebox/hasson/michael/dissertation/test-packrat/packrat/lib-ext' already exists
2: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/test-packr
at/packrat/lib-ext/RCurl', reason 'File exists'
3: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/bitops' to '/jukebox/hasson/michael/dissertation/test-pack
rat/packrat/lib-ext/bitops', reason 'File exists'
4: In symlinkExternalPackages(project = project) :
  The following external packages could not be linked into the packrat private library:
- 'RCurl', 'bitops'
5: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/test-packr
at/packrat/lib-ext/RCurl', reason 'File exists'
6: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/bitops' to '/jukebox/hasson/michael/dissertation/test-pack
rat/packrat/lib-ext/bitops/bitops', reason 'File exists'
7: In symlinkExternalPackages(project = project) :
  The following external packages could not be linked into the packrat private library:
- 'RCurl', 'bitops'
Warning messages:
1: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/RCurl' to '/jukebox/hasson/michael/dissertation/test-packr
at/packrat/lib-ext/RCurl', reason 'Stale NFS file handle'
2: In file.symlink(from, to) :
  cannot symlink '/usr/people/machow/R/library/bitops' to '/jukebox/hasson/michael/dissertation/test-packrat/packrat/lib-ext/bitops', reason 'No such file or directory'
3: In symlinkExternalPackages(project = project) :
  The following external packages could not be linked into the packrat private library:
- 'RCurl', 'bitops'
4: In dir.create(libExtDir(project), recursive = TRUE) :
  '/jukebox/hasson/michael/dissertation/test-packrat/packrat/lib-ext' already exists
kevinushey commented 8 years ago

Hi @machow,

Thanks for the information -- I'll investigate further as well.

kevinushey commented 8 years ago

Hi @machow,

Can you try with the latest development version of packrat once more? It looks like similar work was needed for symlinkExternalPackages(), but the logic there should now be smarter + clean up the accidentally created recursive symlinks.

machow commented 8 years ago

Sorry for the delay--it looks like the symlink cleanup in recent commits has resolved the issue!

ProfFancyPants commented 8 years ago

I think I just had the same issue happen.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Error in ensurePackageSymlink(source, target) : 
  Target 'D:/Dropbox/HIGHSCOPE/packrat/lib-R/mgcv' already exists and is not a symlink

This happened right after opening a 2nd instance of Rstudio. Now it seems to be pretty well stuck on there.

amarchin commented 7 years ago

I got the same problem as well

Error in ensurePackageSymlink(source, target) : 
  Target '/Users/marchiniandrea/Projects/demandr/packrat/lib-R/boot' already exists and is not a symlink

This is my sessionInfo

R version 3.4.1 (2017-06-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.4

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] it_IT.UTF-8/it_IT.UTF-8/it_IT.UTF-8/C/it_IT.UTF-8/it_IT.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.4.1  tools_3.4.1     packrat_0.4.8-1
rentrop commented 7 years ago

any updates?

kevinushey commented 7 years ago

I think this particular issue should be resolved in the development version of Packrat. Can you try installing it and letting me know if the issue still reproduces?

devtools::install_github("rstudio/packrat")
rentrop commented 7 years ago

Unfortunately it does not resolve the issue:

Installing tidyverse (1.1.1) ... OK (downloaded binary) Error in ensurePackageSymlink(source, target) : Target '.../R - onboarding/packrat/lib-R/boot' already exists and is not a symlink

kevinushey commented 7 years ago

@rentrop -- can you try forcefully removing the lib-R directory and trying again?

unlink("packrat/lib-R", recursive = TRUE)

and then restart your R session.

rentrop commented 7 years ago

unlink("packrat/lib-R", recursive = TRUE) devtools::install_github("rstudio/packrat") packrat::init()

Now it "only" throws errors. Any idea how to fix this?

Upgrading these packages already present in packrat:
                 from           to
    BH       1.62.0-1     1.65.0-1
    curl          2.7        2.8.1
    dplyr       0.7.1        0.7.2
    httr        1.2.1        1.3.1
    modelr      0.1.0        0.1.1
    purrr     0.2.2.2        0.2.3
    rlang       0.1.1        0.1.2
    scales      0.4.1   0.5.0.9000
    tibble      1.3.3        1.3.4
    tidyr       0.6.3        0.7.0

Fetching sources for dplyr (0.7.2) ... OK (CRAN current)
Fetching sources for httr (1.3.1) ... OK (CRAN current)
Fetching sources for modelr (0.1.1) ... OK (CRAN current)
Fetching sources for purrr (0.2.3) ... OK (CRAN current)
Fetching sources for scales (0.5.0.9000) ... OK (GitHub)
Fetching sources for tibble (1.3.4) ... OK (CRAN current)
Fetching sources for tidyr (0.7.0) ... OK (CRAN current)
Snapshot written to '/Users/richardrentrop/Sapaso GmbH/Onboarding - Dokumente/R - onboarding/packrat/packrat.lock'
Replacing BH (upgrade 1.62.0-1 to 1.65.0-1) ...     OK (downloaded binary)
Replacing curl (upgrade 2.7 to 2.8.1) ...   OK (downloaded binary)
Replacing rlang (upgrade 0.1.1 to 0.1.2) ...    OK (downloaded binary)
Replacing httr (upgrade 1.2.1 to 1.3.1) ...     OK (downloaded binary)
Replacing tibble (upgrade 1.3.3 to 1.3.4) ...   OK (downloaded binary)
Replacing scales (upgrade 0.4.1 to 0.5.0.9000) ...  OK (built source)
Replacing dplyr (upgrade 0.7.1 to 0.7.2) ...    OK (downloaded binary)
Replacing purrr (upgrade 0.2.2.2 to 0.2.3) ...  OK (downloaded binary)
Replacing tidyr (upgrade 0.6.3 to 0.7.0) ...    OK (downloaded binary)
Replacing modelr (upgrade 0.1.0 to 0.1.1) ...   OK (downloaded binary)
Initialization complete!
Warning messages:
1: The following packages specified in the LinkingTo field for package 'bindrcpp' are unavailable:
- 'plogr'
These packages are required to be installed when attempting to hash this package for caching. 
2: The following packages specified in the LinkingTo field for package 'bindrcpp' are unavailable:
- 'plogr'
These packages are required to be installed when attempting to hash this package for caching. 
3: The following packages specified in the LinkingTo field for package 'dplyr' are unavailable:
- 'BH', 'plogr'
These packages are required to be installed when attempting to hash this package for caching. 
4: The following packages specified in the LinkingTo field for package 'readr' are unavailable:
- 'BH'
These packages are required to be installed when attempting to hash this package for caching. 
5: The following packages specified in the LinkingTo field for package 'xml2' are unavailable:
- 'BH'
These packages are required to be installed when attempting to hash this package for caching. 

Restarting R session...
DiogoFerrari commented 7 years ago

running unlink("packrat/lib-R", recursive = TRUE) and then reopening the R section worked for me. The error happens everytime I change the computer I am working on. So I have to do that over and over again (unlink, reopen R, and then start working). Have anyone found a better solution ?

kendonB commented 6 years ago

I also ran into this problem after reinstalling Windows on my computer. 

Error in ensurePackageSymlink(source, target) : 
  Target 'pathtomyproject/packrat/lib-R/base' already exists and is not a symlink
kendonB commented 6 years ago

@kevinushey should this issue not be reopened?

jekriske-lilly commented 6 years ago

I can confirm this is still an issue as well.

ajmoralesa commented 6 years ago

I am getting the same error when working on the same packrat project from two different machines (Windows and Linux OS)

nick-youngblut commented 6 years ago

I'm getting this issue too:

Error in ensurePackageSymlink(source, target) : 
  Target '/ebio/abt3_projects/Georg_animal_feces/Rproj/packrat/lib-R/cluster' already exists and is not a symlink
Calls: source ... afterPackratModeOn -> symlinkSystemPackages -> ensurePackageSymlink

I'm even getting this error just after restarting my R session:

Restarting R session...

Microsoft R Open 3.4.3
The enhanced R distribution from Microsoft
Microsoft packages Copyright (C) 2017 Microsoft Corporation

Using the Intel MKL for parallel mathematical computing (using 40 cores).

Default CRAN mirror snapshot taken on 2018-01-01.
See: https://mran.microsoft.com/.

Error in ensurePackageSymlink(source, target) : 
  Target '/ebio/abt3_projects/Georg_animal_feces/Rproj/packrat/lib-R/cluster' already exists and is not a symlink

I'm running Rstudio-server on a linux virtual machine (Ubuntu 16.04) with Microsoft R Open. unlink("packrat/lib-R", recursive = TRUE) seems to have fixed the issue, but I just wanted to report that this problem occurred with my specific setup.

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.4 LTS

Matrix products: default
BLAS: /opt/microsoft/ropen/3.4.3/lib64/R/lib/libRblas.so
LAPACK: /opt/microsoft/ropen/3.4.3/lib64/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RevoUtils_10.0.7     RevoUtilsMath_10.0.1

loaded via a namespace (and not attached):
[1] compiler_3.4.3  tools_3.4.3     yaml_2.1.16     packrat_0.4.8-1
nick-youngblut commented 4 years ago

For my R 3.6.2 setup, bioconductor reproducibly creates the packrat symlink.system.packages() error. A reproducible example:

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install(version = "3.10")

The install will fail with a lot of Error in !packrat::opts$symlink.system.packages() : invalid argument type error messages

install.packages() will then not work for any packages. Example:

> install.packages('dplyr')
...
Error in !packrat::opts$symlink.system.packages() : invalid argument type
Calls: source ... setPackratModeOn -> afterPackratModeOn -> symlinkSystemPackages
Execution halted
Warning in install.packages :
  installation of package ‘tidyselect’ had non-zero exit status
Error in !packrat::opts$symlink.system.packages() : invalid argument type
Calls: source ... setPackratModeOn -> afterPackratModeOn -> symlinkSystemPackages
Execution halted
Warning in install.packages :
  installation of package ‘dplyr’ had non-zero exit status
nick-youngblut commented 4 years ago

The issue seems to be associated with the "update packages" portion of the biocManager install. Some of those packages that it's trying to update are in /opt/R/3.6.2/lib/R/library/:

base   cluster    datasets  grDevices   lattice  methods  nnet      spatial  stats4    tools
boot   codetools  foreign   grid        MASS     mgcv     parallel  splines  survival  translations
class  compiler   graphics  KernSmooth  Matrix   nlme     rpart     stats    tcltk     utils

...and these were part of the R-base 3.6.2 install. So, it seems to be a permissions issue associated with biocManager trying to do something with the admin-only files in /opt/R/3.6.2/lib/R/library/

I don't see why nlme and some of these other packages are in the r-base install for v3.6.2. I used the instructions listed here: https://support.rstudio.com/hc/en-us/articles/215488098