Open yogat3ch opened 4 months ago
Can you please provide a reproducible example? Also note that the field is normally called RemoteSha, in case you're generating your renv lock files by hand.
Hi @kevinushey, thank you for the prompt response! Not editing by hand, we use explicit snapshots. I'll see about making a test repo with a Dockerfile, it may take me a bit to get it working properly.
Hi @kevinushey, The repo is up here! To reproduce:
docker build -f Dockerfile -t test:arm64_latest . 2>&1 | tee build.log
And check the build.log
and note the discrepancy in RemoteSha of the UU package.
You can also just take a look at the build.log file that's present from when I just ran it and see that the hash in the log is different from the hash in the renv.lock. It's the hash of the most recent commit on the repo.
Thank you in advance for looking into this!
Thanks -- much appreciated! I'll try to take a closer look tomorrow.
Sounds good, thank you @kevinushey !
Thanks! I've confirmed the issue, and it's specific to usages of GitHub repositories when renv.config.pak.enabled
is set. (To be specific, it's a bug in how renv
is trying to use pak
, not pak
).
We're creating the remote specification using RemoteRef rather than RemoteSha, which is incorrect.
For the curious:
https://github.com/rstudio/renv/blob/5cbf955aef25396ede5d27d5fb375611b4251a4c/R/pak.R#L126-L127
but we prefer using RemoteRef when available when formatting a remote:
https://github.com/rstudio/renv/blob/5cbf955aef25396ede5d27d5fb375611b4251a4c/R/records.R#L131-L138
so we'll want to adjust how we format the remote for use by pak
to prefer using the RemoteSha instead.
Thanks again for providing the reproducible example; it made it far easier to ascertain the root cause here!
@kevinushey Awesome, thank you for diagnosing the issue! So is it just a matter of switching the order of the if else
statement there?
Not quite; that function is used both when trying to format a record as a remote for display (where we might want to truncate the SHA since just the first few characters are probably fine), whereas we'd want to preserve the entire SHA when using that for an actual installation request. I'm going to poke at it a bit further.
This should now be fixed up in the development version of renv
-- would you be able to test and verify on your side as well?
Hi @kevinushey, Verified working on my end! Thank you for fixing that so quickly!
Hi @kevinushey, It looks like this is having some unexpected downstream consequences with pak. It seems like pak has some kind of internal dependency version checking for when multiple packages point to different versions of a single dependency, it has no way of resolving which to install
#19 49.58 Error:
--
2478 | #19 49.58 ! error in pak subprocess
2479 | #19 49.58 Caused by error:
2480 | #19 49.58 ! Could not solve package dependencies:
2481 | #19 49.58 * yogat3ch/DTExtend@436c239d6c7c7c4bc4df1d7ff2acc9b03fb0934a: Can't install dependency yogat3ch/UU (>= 1.44.0)
2482 | #19 49.58 * yogat3ch/UU: Conflicts with yogat3ch/UU@1935fc3262898ed6c93237407565e3b910579616
2483 | #19 49.58 * yogat3ch/OpenMCE@a1ad9b528138a58b0537531aa8dcc00be12e7850: Can't install dependency yogat3ch/UU
2484 | #19 49.58 * yogat3ch/bs4Dash@1d4d4618e47457067fd07e17c840e58f0213c684: Can't install dependency yogat3ch/UU (>= 1.5.1)
2485 | #19 49.58 * Martin-McCoy/DMDUAPI@3846630ba715eef40230ae5e81608dc129bbac32: Can't install dependency yogat3ch/UU (>= 1.39.3)
2486 | #19 49.58 * nlme@3.1-162: dependency conflict
2487 | #19 49.58 * Martin-McCoy/shinyVirga@59764669b9b9fe679220c78130e038fd548d0231: Can't install dependency yogat3ch/UU (>= 1.43.0)
2488 | #19 49.58 * JohnCoene/tippy@0d89b2a5cf67c7ba2868a4777125326ef462ec76: dependency conflict
2489 | #19 49.58 * Martin-McCoy/virgaUtils@3dbd35799aa988f6b43ebe512a64069f77356e2c: Can't install dependency yogat3ch/UU (>= 0.24.2)
2490 | #19 49.58 * codetools@0.2-19: dependency conflict
2491 | #19 49.58 * R6@2.5.1: dependency conflict
2492 | #19 49.58 * cpp11@0.4.7: dependency conflict
2493 | #19 49.58 * credentials@2.0.1: dependency conflict
2494 | #19 49.58 * curl@5.2.1: dependency conflict
2495 | #19 49.58 * dplyr@1.1.4: dependency conflict
Does it make sense to you that this would happen or should I try to create a reprex?
It seems like it needs to resolve to whichever is the latest, unless that conflict with an explicit exact version/hash declaration that uses the ==
?
Or it could just ignore all the dependency versions and install whatever the explicitly declared version is in the renv.lock
, and if there isn't one, encourage the user to install the version they want and snapshot it.
Hey @kevinushey,
Bumping this as the workaround we tried does not fix the issue. renv
will use the latest version of every Github package if the package is installed from Github, even if the version number is not what's specified in the renv.lock and even if the DESCRIPTION explicitly specifies a package version other than the most recent.
A quick scan over the recent issues suggests that 9328cfe might be causing other people's builds to fail if they installed renv
from Github, given how renv
is automatically using the most recent version of everything installed from Github, anyone with renv
recorded in their lockfile from Github is going to get the latest version, regardless of whats in the lockfile.
We just built these two utilities for overwriting the latest version that renv
installs with the version specified in the lockfile. Hopefully they prove helpful for fixing the issue:
renv_gh_pkgs <- function(file = "renv.lock") {
lf <- if (!inherits(file, "renv_lockfile"))
renv::lockfile_read(file)
else
file
gh_pkgs <- purrr::keep(lf$Packages, \(.x) .x$Source == "GitHub")
}
renv_gh_pkgs_install <- function(pkgs) {
purrr::walk(pkgs, \(.x) {
remotes::install_github(
sprintf("%s/%s", .x$RemoteUsername, .x$RemoteRepo),
ref = .x$RemoteSha
)
})
}
We've implemented after renv::restore
in the Dockerfile as follows:
# Copies the renv.lock file from your local project into the Docker image.
# This lock file ensures that R packages are installed at specific versions for reproducibility.
COPY renv.lock renv.lock
# Copies the .Renviron file, which typically contains environment variables for R.
# This can include API keys or other credentials needed for your R environment.
COPY .Renviron .Renviron
# Copies the .Rprofile, which can contain R commands to run every time an R session starts.
# This can be used to set global options or load certain packages by default.
COPY .Rprofile .Rprofile
# Copies the .Rbuildignore file into the Docker image.
# This file indicates which files and directories should be ignored by R build tools.
COPY .Rbuildignore .Rbuildignore
# Copies the 'renv' directory, which contains R package caches and configuration files.
# This directory helps 'renv' manage packages more efficiently.
COPY renv renv
# Sets the RENV_PATHS_LIBRARY environment variable.
# This tells 'renv' where to store and look for installed R packages.
ENV RENV_PATHS_LIBRARY /usr/local/lib/R/site-library
# Installs the 'remotes' package, which is useful for installing R packages from sources such as GitHub.
RUN R --quiet -e "install.packages('remotes')"
# Installs the 'renv' package, a package management tool that helps isolate project-specific dependencies.
# This version includes a fix that ensures the SHA of Github packages recorded in the lockfile are installed instead of the most recent version.
RUN R --quiet -e "install.packages('renv')"
# Restores the R environment based on the renv.lock file.
# This ensures that all R package dependencies are installed.
RUN R --quiet -e "renv::diagnostics();renv::restore(library = '/usr/local/lib/R/site-library')"
# Copy functions for installing specific versions
COPY R/utils_renv.R R/utils_renv.R
# Overwrite whatever versions the `renv::restore` installed for Github Repositories with those recorded in the lockfile
RUN R --quiet -e "source('R/utils_renv.R'); renv_gh_pkgs_install(renv_gh_pkgs());"
Could you also share the lockfile where you saw this issue, so I can try to reproduce locally and better understand the issue?
Hi @kevinushey,
I've updated the testRenvDocker
repo with a lockfile that is similar to the one we're using except that it has private Github repos removed. I've installed the dev version of renv
there so It will throw the dependency conflict error mentioned above.
As I'm debugging this Dockerfile, it looks like renv
is not observing version numbers for CRAN installed packages. It seems like the most recent version of DT gets installed as a dependency, renv::restore
detects the presence of this later version of DT and doesn't install the version specified in the lockfile.
It seems like renv::restore
needs an argument like use_lockfile_versions
that will re-install the specific Github hashes and or versions of packages specified by the lockfile. rebuild
is close, but that sounds like it reinstalls literally everything, and not just the packages that have Version/RemoteSha discrepancies. Thoughts on that approach?
We've happened upon another issue, which I can move over to another issue if need be. UPDATE The below issue is resolved. Rebooting the computer was apparently all that was needed since that's the only change since I tried last night! 🤷
We re-arranged our workaround a bit in the Dockerfile:
# This tells 'renv' where to store and look for installed R packages.
ENV RENV_PATHS_LIBRARY /usr/local/lib/R/site-library
# Installs the 'remotes' package, which is useful for installing R packages from sources such as GitHub.
RUN R --quiet -e "install.packages('remotes', lib = Sys.getenv('RENV_PATHS_LIBRARY'))"
# Installs the 'renv' package, a package management tool that helps isolate project-specific dependencies.
# This version includes a fix that ensures the SHA of Github packages recorded in the lockfile are installed instead of the most recent version.
RUN R --quiet -e "install.packages('renv', lib = Sys.getenv('RENV_PATHS_LIBRARY'))"
# Copy functions for installing specific versions
COPY R/utils_renv.R R/utils_renv.R
# Need purrr for subsequent calls
RUN R --quiet -e "install.packages('purrr', lib = Sys.getenv('RENV_PATHS_LIBRARY'))"
# Overwrite whatever versions the `renv::restore` installed for Github Repositories with those recorded in the lockfile
RUN R --quiet -e "source('R/utils_renv.R'); renv::deactivate(); renv_gh_pkgs_install(renv_gh_pkgs());"
# Restores the R environment based on the renv.lock file.
# This ensures that all R package dependencies are installed.
RUN R --quiet -e "renv::deactivate();source('R/utils_renv.R'); renv::restore(exclude = names(renv_gh_pkgs()))"
In order to install the Github repos first, using their specific RemoteSha values, and then exclude them from the renv::restore
so as not to install the dependencies twice.
For some reason, switching the order of operations has now produced this obtuse error when running renv::restore
:
#21 146.7 ℹ Installing system requirements
#21 146.7 ℹ Executing `sh -c apt-get -y update`
#21 150.7 Error:
#21 150.7 ! error in pak subprocess
#21 150.7 Caused by error in `processx::run(sh, cmdline, stdout_callback = callback, stderr_to_stdout = TRUE)`:
#21 150.7 ! System command 'sh' failed
The container is built off of this image:
FROM rocker/r-ver:4.3.0@sha256:9c1703e265fca5a17963a1b255b3b2ead6dfc6d65c57e4af2f31bec15554da86 AS base-arm64
And I verified that it can run sh
commands by booting an already built version of it interactively using the sh
command
Any idea what's going on here? Seems like it can't run the sh
command from the background process initialized by processx::run
? Should I move this to a new issue?
Hey @kevinushey, We're still trying to figure out a workaround for this issue as it's holding up our production build. Having the right versions of packages is critical.
In the process of attempting to derive a functioning workaround we're encountering other unexpected issues, one of which I thought ought to be surfaced to the renv
team.
I'm interactively debugging a container built from the rocker/r-ver:4.3.0@sha256:9c1703e265fca5a17963a1b255b3b2ead6dfc6d65c57e4af2f31bec15554da86
ARM64 build.
The container has been built with the 2.2.1 version of bs4Dash
, which can be seen in the packageDescription
output below:
packageDescription("bs4Dash")
Package: bs4Dash
Type: Package
Title: A 'Bootstrap 4' Version of 'shinydashboard'
Version: 2.2.1
Authors@R: c( person("David", "Granjon", email = "dgranjon@ymail.com",
role = c("aut", "cre")), person(family = "RinteRface", role =
"cph"), person(family = "Almasaeed Studio", role = c("ctb",
"cph"), comment = "AdminLTE3 theme for Bootstrap 4"),
person("Winston", "Chang", role = c("ctb", "cph"), comment =
"Utils functions from shinydashboard"))
Maintainer: David Granjon <dgranjon@ymail.com>
Description: Make 'Bootstrap 4' Shiny dashboards. Use the full power of
'AdminLTE3', a dashboard template built on top of 'Bootstrap 4'
<https://github.com/ColorlibHQ/AdminLTE>.
URL: https://rinterface.github.io/bs4Dash/index.html,
https://github.com/RinteRface/bs4Dash
BugReports: https://github.com/RinteRface/bs4Dash/issues
License: GPL (>= 2) | file LICENSE
Imports: shiny (>= 1.6.0), htmltools (>= 0.5.1.1), jsonlite (>=
0.9.16), fresh, grDevices, waiter (>= 0.2.3), httpuv (>=
1.5.2), lifecycle, bslib (>= 0.2.4), httr, UU (>= 1.5.1)
Suggests: knitr, rmarkdown, testthat (>= 2.1.0), golem, DT, thematic
(>= 0.1.2)
Encoding: UTF-8
RoxygenNote: 7.2.3
VignetteBuilder: knitr
Collate: 'aaa_reimports.R' 'feedbacks.R' 'useful-items.R' 'tabs.R'
.....
RdMacros: lifecycle
Remotes: yogat3ch/UU
NeedsCompilation: no
Packaged: 2024-05-23 14:58:06 UTC; root
Author: David Granjon [aut, cre], RinteRface [cph], Almasaeed Studio
[ctb, cph] (AdminLTE3 theme for Bootstrap 4), Winston Chang
[ctb, cph] (Utils functions from shinydashboard)
Built: R 4.3.0; ; 2024-05-23 14:58:07 UTC; unix
RemoteType: github
RemoteHost: api.github.com
RemoteRepo: bs4Dash
RemoteUsername: yogat3ch
RemotePkgRef: yogat3ch/bs4Dash@HEAD
RemoteRef: HEAD
RemoteSha: 1d4d4618e47457067fd07e17c840e58f0213c684
GithubRepo: bs4Dash
GithubUsername: yogat3ch
GithubRef: HEAD
GithubSHA1: 1d4d4618e47457067fd07e17c840e58f0213c684
-- File: /usr/local/lib/R/site-library/bs4Dash/Meta/package.rds
As part of the Dockerfile build process, one step is renv::restore
which produces the following output:
RUN R --quiet -e "renv::restore()"
#10 [stage-4 5/5] RUN R --quiet -e "renv::restore()"
#10 0.368 > renv::restore()
#10 4.992
#10 6.615 ✔ Updated metadata database: 1.37 MB in 1 file.
#10 6.615
#10 6.616 ℹ Updating metadata database
#10 8.851 ✔ Updating metadata database ... done
#10 8.852
#10 9.077
#10 9.078
#10 9.240 ℹ No downloads are needed
#10 9.273 ✔ 2 pkgs + 1 dep: [5.3s]
#10 9.296 >
#10 9.296 >
#10 DONE 9.8s
despite the fact that in the renv.lock
has the following entry (viewed from the output of renv::lockfile_read
):
> rl$Packages$bs4Dash
$Package
[1] "bs4Dash"
$Version
[1] "2.3.3"
$Source
[1] "GitHub"
$Remotes
[1] "yogat3ch/UU"
$RemoteType
[1] "github"
$RemoteHost
[1] "api.github.com"
$RemoteRepo
[1] "bs4Dash"
$RemoteUsername
[1] "yogat3ch"
$RemoteRef
[1] "bc1e40f"
$RemoteSha
[1] "bc1e40f404915e566c0a70acc543288a0928989f"
$Requirements
[1] "UU" "bslib" "fresh" "grDevices" "htmltools" "httpuv"
[7] "httr" "jsonlite" "lifecycle" "shiny" "waiter"
$Hash
[1] "a707c272226b7265b5d2221f66a604c8"
There's a discrepancy in version number and hash number that renv::restore
doesn't seem to pick up on.
I've replicated this same behavior outside the Docker container on my Mac environment.
This is the output of .libPaths
for context:
.libPaths()
[1] "/Users/stephenholsenbeck/Library/Caches/org.R-project.R/R/renv/library/dmdu-4393acb2/R-4.3/aarch64-apple-darwin20"
[2] "/Users/stephenholsenbeck/Library/Caches/org.R-project.R/R/renv/sandbox/R-4.3/aarch64-apple-darwin20/ac5c2659"
Am I missing something critical about how renv::restore
is supposed to work? I thought it was meant to synchronize the version of packages in the library with the versions of packages specified in the lockfile? I feel like renv::restore
has synchronized based on the lockfile properly in the past when there was a version discrepancy, but it doesn't seem to be doing it anymore? Or is there some setting that's making it overlook the discrepancy in package versions?
Hi @kevinushey and renv dev team, We're using
renv::restore
as part of our Docker build process and just discovered that whenrenv::restore
runs while building the Docker image it's not respecting the RemoteSHA recorded in the lockfile for github dependencies, and is rather using the most recent commit. I was under the assumption thatrenv::restore
uses the RemoteSHA specified in the renv.lock file, but that doesn't seem to be the case. Am I missing a configuration option to ensure thatrenv::restore
uses the RemoteSHA from the lock file?