Open krlmlr opened 1 year ago
Attaching an example renv.lock
file, to give you an idea about the pain I went through today. Restoring this on an M1 for R 4.1 doesn't work, because some packages are only available for slightly younger versions on arm64. Unfortunately, a plain restore()
attempts to download all packages, and then fails at the first occasion.
Just to add that for me this is currently the biggest pain point with using renv, particularly across systems:
restore()
=> package fails because of availability or compilation => install binary/newer version, record => restore()
again" loop is very long and needs to be run multiple times, particularly on systems with missing build tools (e.g. Windows machine at work). The rechecking of all packages at each retry (with all Github repos being redownloaded) makes this even longer.I am not sure what a good solution would be, but I can imagine a flow where
restore()
installs everything that can be installed given unfound/uncompilable packagesAn alternative solution would be a function to update the lockfile such that it captures either latest binaries or binaries closest to the snapshotted ones.
In the meantime, a simple way of updating older lockfiles to use P3M would help with some of this by making it easy to access binaries for older package versions.
I agree overall that renv
could do more to ease the process here.
An alternative solution would be a function to update the lockfile such that it captures either latest binaries or binaries closest to the snapshotted ones.
Wouldn't this be the same as just calling renv::install()
to install the latest-required versions of packages for a project?
In the meantime, a simple way of updating older lockfiles to use P3M would help with some of this by making it easy to access binaries for older package versions.
There's a few ways you could accomplish this:
lockfile_modify()
+ lockfile_write()
to update the repositories in the lockfile,renv::restore(repos = <...>)
to specify the repositories at restore time explicitly.Were you hoping for something more streamlined?
Many thanks @kevinushey. I also should have said renv is great overall!
Wouldn't this be the same as just calling renv::install() to install the latest-required versions of packages for a project?
I think what I was looking for was this to happen en masse, i.e. for renv to go through the lockfile, compare with availability of binaries in repo (ideally P3M - see below), and update the lockfile. Doing it one by one is possible but at some size of lockfile becomes draining.
There's a few ways you could accomplish this:
Thanks - will try. I think what would help would be a simple step to update a lockfile created in renv < 1.0.X such that it works with P3M the same way as a lockfile in a project initialised using renv > v1.0.X, without me having to figure out how to define the reference to P3M in the repo
entry. The catch is that this is more difficult to do by re-initiatising a project when the project is in an inconsistent state (unrestored) e.g. when one comes back to a project after a while that was last used with renv < v1. (Perhaps this could be an argument to renv::upgrade()
?)
In hindsight I think what I am unclear about is how to reference P3M such that renv
can search for previous versions in it, so it works like CRAN but with binaries stored for past versions (rather than a snapshot from a given date etc.)
I understand all this is mostly caused by missing build tools which the user can ultimately fix, but I suspect it is quite a common situation.
In hindsight I think what I am unclear about is how to reference P3M such that renv can search for previous versions in it, so it works like CRAN but with binaries stored for past versions (rather than a snapshot from a given date etc.)
The latest release of renv
should be doing this by default on Windows + macOS -- for example, I see:
> renv::install("dplyr@1.1.2")
# Downloading packages -------------------------------------------------------
- Querying repositories for available binary packages ... Done!
- Downloading dplyr from P3M ... OK [1.4 Mb in 2.5s]
Successfully downloaded 1 package in 4.9 seconds.
The following package(s) will be installed:
- dplyr [1.1.2]
These packages will be installed into "~/Library/R/arm64/4.3/library".
Do you want to proceed? [Y/n]: y
# Installing packages --------------------------------------------------------
- Installing dplyr ... OK [installed binary and cached in 0.62s]
Successfully installed 1 package in 0.73 seconds.
This behavior is controlled via the configuration options here, but should be on by default:
https://rstudio.github.io/renv/reference/config.html#renv-config-ppm-enabled
@kevinushey could you please describe the workflow you have in mind, in particular whether/how the packages
or exclude
arguments can fulfill the "keep going" feature suggested by @krlmlr ?
For now, my (very cumbersome) workflow is:
renv::restore()
renv::record()
This workflow is really problematic, as the packages that were successfully installed with the previous renv::restore()
are still re-installed, so restoring a single project can easily take 4-10 hours.
This seems also at odd with the documentation, which I interpret as saying that only missing/non-installed packages are going to be installed? But do I understand you correctly that the packages
or exclude
arguments can help avoid this?
renv::restore() compares packages recorded in the lockfile to the packages installed in the project library. Where there are differences it resolves them by installing the lockfile-recorded package into the project library.
and by the way, the same could be said about downloading the packages, which seem to happen at each restore()
call, even if packages have been already downloaded?
This workflow is really problematic, as the packages that were successfully installed with the previous renv::restore() are still re-installed, so restoring a single project can easily take 4-10 hours.
I believe if you set:
options(renv.config.install.transactional = FALSE)
then you'll see the behavior you're hoping for; that is, any packages which are successfully installed during renv::restore()
will remain in your private library. For example, I tried making a local package called help
which depends on rlang
, but fails to install:
> renv::install("~/scratch/help")
The following package(s) will be installed:
- help [0.0.0.9000]
- rlang [1.1.4]
These packages will be installed into "~/scratch/test/renv/library/macos/R-4.4/aarch64-apple-darwin20".
Do you want to proceed? [Y/n]: y
# Installing packages --------------------------------------------------------
- Installing rlang ... OK [linked from cache]
- Installing help ... FAILED
Error: Error installing package 'help':
================================
* installing *source* package ‘help’ ...
** using staged installation
** R
** byte-compile and prepare package for lazy loading
** help
No man pages found in package ‘help’
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘help’:
.onLoad failed in loadNamespace() for 'help', details:
call: fun(libname, pkgname)
error: Oh no
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Users/kevin/scratch/test/renv/library/macos/R-4.4/aarch64-apple-darwin20/help’
install of package 'help' failed [error code 1]
Traceback (most recent calls last):
11: renv::install("~/scratch/help")
10: renv_install_impl(records) at install.R#231
9: if (staged)
renv_install_staged(records)
else
renv_install_default(records) at install.R#275
8: handler(package, renv_install_package(record)) at install.R#398
7: handler(package, renv_install_package(record)) at install.R#398
6: withCallingHandlers(
renv_install_package_impl(record),
error = function(e) writef("FAILED")
) at install.R#431
5: withCallingHandlers(
renv_install_package_impl(record),
error = function(e) writef("FAILED")
) at install.R#431
4: if (copyable)
renv_file_copy(path, installpath, overwrite = TRUE)
else
r_cmd_install(package, path) at install.R#641
3: if (!identical(status, 0L))
r_exec_error(package, output, "install", status) at r.R#234
2: abort(all) at r.R#52
1: stop(fallback) at abort.R#44
> find.package("rlang")
[1] "/Users/kevin/scratch/test/renv/library/macos/R-4.4/aarch64-apple-darwin20/rlang"
The intention of the default being TRUE
is that we didn't want renv::restore()
to leave the project library in an inconsistent state on failure. This is less relevant if someone is trying to use renv::restore()
multiple times due to some package(s) failing to install, though.
I wonder if making this option more visible via:
renv::restore(transactional = FALSE)
would be worthwhile.
This is great news!!!! Absolutely, putting this argument at the forefront would be very beneficial (you can't imagine how many hours were lost without this option). Maybe could think of a more explicit name, such as dont_restart
or keep_installed
?
Now that said, I realize that my comments were really about this "keep installed" strategy, and not the "keep going" which this post is all about, sorry for hijacking (though it feels the "keep installed" is a necessary step for the "keep going"?)
I agree this is a very important feature to add. After each failed package installation (often due to being too old), I install the latest version and run renv::record()
to update just that package and not remove those not installed yet via renv::snapshot()
, then another run of renv::restore()
until the next fail. Each iteration I'd have to reinstall all the successfully installed packages. Having transactional = FALSE
option in renv::restore()
would be great.
Follow-up to #746. A perhaps more ambitious variant of #1108.
When attempting
renv::restore()
from a largerenv.lock
file, therenv.config.install.transactional
config already avoids a lot of repeated work. Ideally, a "keep going" mode, similar tomake -k
, would attempt to install all packages that can be installed (and even avoid erroring on download failures). The firstrestore()
would then be a "best effort" install, following attempts would focus on installing packages that failed with the first attempt (in combination withrenv.config.install.transactional
).