kevinushey
commented
4 months ago Can you please provide a reproducible example? Also note that the field is normally called RemoteSha, in case you're generating your renv lock files by hand.
Open yogat3ch opened 4 months ago
kevinushey
commented
4 months ago Can you please provide a reproducible example? Also note that the field is normally called RemoteSha, in case you're generating your renv lock files by hand.
yogat3ch
commented
4 months ago Hi @kevinushey, thank you for the prompt response! Not editing by hand, we use explicit snapshots. I'll see about making a test repo with a Dockerfile, it may take me a bit to get it working properly.
yogat3ch
commented
4 months ago Hi @kevinushey, The repo is up here! To reproduce:
docker build -f Dockerfile -t test:arm64_latest . 2>&1 | tee build.logAnd check the build.log and note the discrepancy in RemoteSha of the UU package.
You can also just take a look at the build.log file that's present from when I just ran it and see that the hash in the log is different from the hash in the renv.lock. It's the hash of the most recent commit on the repo.
Thank you in advance for looking into this!
kevinushey
commented
4 months ago Thanks -- much appreciated! I'll try to take a closer look tomorrow.
yogat3ch
commented
4 months ago Sounds good, thank you @kevinushey !
kevinushey
commented
4 months ago Thanks! I've confirmed the issue, and it's specific to usages of GitHub repositories when renv.config.pak.enabled is set. (To be specific, it's a bug in how renv is trying to use pak, not pak).
We're creating the remote specification using RemoteRef rather than RemoteSha, which is incorrect.
For the curious:
https://github.com/rstudio/renv/blob/5cbf955aef25396ede5d27d5fb375611b4251a4c/R/pak.R#L126-L127
but we prefer using RemoteRef when available when formatting a remote:
https://github.com/rstudio/renv/blob/5cbf955aef25396ede5d27d5fb375611b4251a4c/R/records.R#L131-L138
so we'll want to adjust how we format the remote for use by pak to prefer using the RemoteSha instead.
Thanks again for providing the reproducible example; it made it far easier to ascertain the root cause here!
yogat3ch
commented
4 months ago @kevinushey Awesome, thank you for diagnosing the issue! So is it just a matter of switching the order of the if else statement there?
kevinushey
commented
4 months ago Not quite; that function is used both when trying to format a record as a remote for display (where we might want to truncate the SHA since just the first few characters are probably fine), whereas we'd want to preserve the entire SHA when using that for an actual installation request. I'm going to poke at it a bit further.
kevinushey
commented
4 months ago This should now be fixed up in the development version of renv -- would you be able to test and verify on your side as well?
yogat3ch
commented
4 months ago Hi @kevinushey, Verified working on my end! Thank you for fixing that so quickly!
yogat3ch
commented
3 months ago Hi @kevinushey, It looks like this is having some unexpected downstream consequences with pak. It seems like pak has some kind of internal dependency version checking for when multiple packages point to different versions of a single dependency, it has no way of resolving which to install
#19 49.58 Error:
--
2478 | #19 49.58 ! error in pak subprocess
2479 | #19 49.58 Caused by error:
2480 | #19 49.58 ! Could not solve package dependencies:
2481 | #19 49.58 * yogat3ch/DTExtend@436c239d6c7c7c4bc4df1d7ff2acc9b03fb0934a: Can't install dependency yogat3ch/UU (>= 1.44.0)
2482 | #19 49.58 * yogat3ch/UU: Conflicts with yogat3ch/UU@1935fc3262898ed6c93237407565e3b910579616
2483 | #19 49.58 * yogat3ch/OpenMCE@a1ad9b528138a58b0537531aa8dcc00be12e7850: Can't install dependency yogat3ch/UU
2484 | #19 49.58 * yogat3ch/bs4Dash@1d4d4618e47457067fd07e17c840e58f0213c684: Can't install dependency yogat3ch/UU (>= 1.5.1)
2485 | #19 49.58 * Martin-McCoy/DMDUAPI@3846630ba715eef40230ae5e81608dc129bbac32: Can't install dependency yogat3ch/UU (>= 1.39.3)
2486 | #19 49.58 * nlme@3.1-162: dependency conflict
2487 | #19 49.58 * Martin-McCoy/shinyVirga@59764669b9b9fe679220c78130e038fd548d0231: Can't install dependency yogat3ch/UU (>= 1.43.0)
2488 | #19 49.58 * JohnCoene/tippy@0d89b2a5cf67c7ba2868a4777125326ef462ec76: dependency conflict
2489 | #19 49.58 * Martin-McCoy/virgaUtils@3dbd35799aa988f6b43ebe512a64069f77356e2c: Can't install dependency yogat3ch/UU (>= 0.24.2)
2490 | #19 49.58 * codetools@0.2-19: dependency conflict
2491 | #19 49.58 * R6@2.5.1: dependency conflict
2492 | #19 49.58 * cpp11@0.4.7: dependency conflict
2493 | #19 49.58 * credentials@2.0.1: dependency conflict
2494 | #19 49.58 * curl@5.2.1: dependency conflict
2495 | #19 49.58 * dplyr@1.1.4: dependency conflict
Does it make sense to you that this would happen or should I try to create a reprex?
It seems like it needs to resolve to whichever is the latest, unless that conflict with an explicit exact version/hash declaration that uses the == ?
Or it could just ignore all the dependency versions and install whatever the explicitly declared version is in the renv.lock, and if there isn't one, encourage the user to install the version they want and snapshot it.
yogat3ch
commented
3 months ago Hey @kevinushey,
Bumping this as the workaround we tried does not fix the issue. renv will use the latest version of every Github package if the package is installed from Github, even if the version number is not what's specified in the renv.lock and even if the DESCRIPTION explicitly specifies a package version other than the most recent.
A quick scan over the recent issues suggests that 9328cfe might be causing other people's builds to fail if they installed renv from Github, given how renv is automatically using the most recent version of everything installed from Github, anyone with renv recorded in their lockfile from Github is going to get the latest version, regardless of whats in the lockfile.
yogat3ch
commented
3 months ago We just built these two utilities for overwriting the latest version that renv installs with the version specified in the lockfile. Hopefully they prove helpful for fixing the issue:
renv_gh_pkgs <- function(file = "renv.lock") {
lf <- if (!inherits(file, "renv_lockfile"))
renv::lockfile_read(file)
else
file
gh_pkgs <- purrr::keep(lf$Packages, \(.x) .x$Source == "GitHub")
}
renv_gh_pkgs_install <- function(pkgs) {
purrr::walk(pkgs, \(.x) {
remotes::install_github(
sprintf("%s/%s", .x$RemoteUsername, .x$RemoteRepo),
ref = .x$RemoteSha
)
})
}We've implemented after renv::restore in the Dockerfile as follows:
# Copies the renv.lock file from your local project into the Docker image.
# This lock file ensures that R packages are installed at specific versions for reproducibility.
COPY renv.lock renv.lock
# Copies the .Renviron file, which typically contains environment variables for R.
# This can include API keys or other credentials needed for your R environment.
COPY .Renviron .Renviron
# Copies the .Rprofile, which can contain R commands to run every time an R session starts.
# This can be used to set global options or load certain packages by default.
COPY .Rprofile .Rprofile
# Copies the .Rbuildignore file into the Docker image.
# This file indicates which files and directories should be ignored by R build tools.
COPY .Rbuildignore .Rbuildignore
# Copies the 'renv' directory, which contains R package caches and configuration files.
# This directory helps 'renv' manage packages more efficiently.
COPY renv renv
# Sets the RENV_PATHS_LIBRARY environment variable.
# This tells 'renv' where to store and look for installed R packages.
ENV RENV_PATHS_LIBRARY /usr/local/lib/R/site-library
# Installs the 'remotes' package, which is useful for installing R packages from sources such as GitHub.
RUN R --quiet -e "install.packages('remotes')"
# Installs the 'renv' package, a package management tool that helps isolate project-specific dependencies.
# This version includes a fix that ensures the SHA of Github packages recorded in the lockfile are installed instead of the most recent version.
RUN R --quiet -e "install.packages('renv')"
# Restores the R environment based on the renv.lock file.
# This ensures that all R package dependencies are installed.
RUN R --quiet -e "renv::diagnostics();renv::restore(library = '/usr/local/lib/R/site-library')"
# Copy functions for installing specific versions
COPY R/utils_renv.R R/utils_renv.R
# Overwrite whatever versions the `renv::restore` installed for Github Repositories with those recorded in the lockfile
RUN R --quiet -e "source('R/utils_renv.R'); renv_gh_pkgs_install(renv_gh_pkgs());"
Could you also share the lockfile where you saw this issue, so I can try to reproduce locally and better understand the issue?
yogat3ch
commented
3 months ago Hi @kevinushey,
I've updated the testRenvDocker repo with a lockfile that is similar to the one we're using except that it has private Github repos removed. I've installed the dev version of renv there so It will throw the dependency conflict error mentioned above.
As I'm debugging this Dockerfile, it looks like renv is not observing version numbers for CRAN installed packages. It seems like the most recent version of DT gets installed as a dependency, renv::restore detects the presence of this later version of DT and doesn't install the version specified in the lockfile.
It seems like renv::restore needs an argument like use_lockfile_versions that will re-install the specific Github hashes and or versions of packages specified by the lockfile. rebuild is close, but that sounds like it reinstalls literally everything, and not just the packages that have Version/RemoteSha discrepancies. Thoughts on that approach?
We've happened upon another issue, which I can move over to another issue if need be. UPDATE The below issue is resolved. Rebooting the computer was apparently all that was needed since that's the only change since I tried last night! 🤷
We re-arranged our workaround a bit in the Dockerfile:
# This tells 'renv' where to store and look for installed R packages.
ENV RENV_PATHS_LIBRARY /usr/local/lib/R/site-library
# Installs the 'remotes' package, which is useful for installing R packages from sources such as GitHub.
RUN R --quiet -e "install.packages('remotes', lib = Sys.getenv('RENV_PATHS_LIBRARY'))"
# Installs the 'renv' package, a package management tool that helps isolate project-specific dependencies.
# This version includes a fix that ensures the SHA of Github packages recorded in the lockfile are installed instead of the most recent version.
RUN R --quiet -e "install.packages('renv', lib = Sys.getenv('RENV_PATHS_LIBRARY'))"
# Copy functions for installing specific versions
COPY R/utils_renv.R R/utils_renv.R
# Need purrr for subsequent calls
RUN R --quiet -e "install.packages('purrr', lib = Sys.getenv('RENV_PATHS_LIBRARY'))"
# Overwrite whatever versions the `renv::restore` installed for Github Repositories with those recorded in the lockfile
RUN R --quiet -e "source('R/utils_renv.R'); renv::deactivate(); renv_gh_pkgs_install(renv_gh_pkgs());"
# Restores the R environment based on the renv.lock file.
# This ensures that all R package dependencies are installed.
RUN R --quiet -e "renv::deactivate();source('R/utils_renv.R'); renv::restore(exclude = names(renv_gh_pkgs()))"
In order to install the Github repos first, using their specific RemoteSha values, and then exclude them from the renv::restore so as not to install the dependencies twice.
For some reason, switching the order of operations has now produced this obtuse error when running renv::restore:
#21 146.7 ℹ Installing system requirements
#21 146.7 ℹ Executing `sh -c apt-get -y update`
#21 150.7 Error:
#21 150.7 ! error in pak subprocess
#21 150.7 Caused by error in `processx::run(sh, cmdline, stdout_callback = callback, stderr_to_stdout = TRUE)`:
#21 150.7 ! System command 'sh' failedThe container is built off of this image:
FROM rocker/r-ver:4.3.0@sha256:9c1703e265fca5a17963a1b255b3b2ead6dfc6d65c57e4af2f31bec15554da86 AS base-arm64And I verified that it can run sh commands by booting an already built version of it interactively using the sh command
Any idea what's going on here? Seems like it can't run the sh command from the background process initialized by processx::run? Should I move this to a new issue?
yogat3ch
commented
3 months ago Hey @kevinushey, We're still trying to figure out a workaround for this issue as it's holding up our production build. Having the right versions of packages is critical.
In the process of attempting to derive a functioning workaround we're encountering other unexpected issues, one of which I thought ought to be surfaced to the renv team.
I'm interactively debugging a container built from the rocker/r-ver:4.3.0@sha256:9c1703e265fca5a17963a1b255b3b2ead6dfc6d65c57e4af2f31bec15554da86 ARM64 build.
The container has been built with the 2.2.1 version of bs4Dash, which can be seen in the packageDescription output below:
packageDescription("bs4Dash")
Package: bs4Dash
Type: Package
Title: A 'Bootstrap 4' Version of 'shinydashboard'
Version: 2.2.1
Authors@R: c( person("David", "Granjon", email = "dgranjon@ymail.com",
role = c("aut", "cre")), person(family = "RinteRface", role =
"cph"), person(family = "Almasaeed Studio", role = c("ctb",
"cph"), comment = "AdminLTE3 theme for Bootstrap 4"),
person("Winston", "Chang", role = c("ctb", "cph"), comment =
"Utils functions from shinydashboard"))
Maintainer: David Granjon <dgranjon@ymail.com>
Description: Make 'Bootstrap 4' Shiny dashboards. Use the full power of
'AdminLTE3', a dashboard template built on top of 'Bootstrap 4'
<https://github.com/ColorlibHQ/AdminLTE>.
URL: https://rinterface.github.io/bs4Dash/index.html,
https://github.com/RinteRface/bs4Dash
BugReports: https://github.com/RinteRface/bs4Dash/issues
License: GPL (>= 2) | file LICENSE
Imports: shiny (>= 1.6.0), htmltools (>= 0.5.1.1), jsonlite (>=
0.9.16), fresh, grDevices, waiter (>= 0.2.3), httpuv (>=
1.5.2), lifecycle, bslib (>= 0.2.4), httr, UU (>= 1.5.1)
Suggests: knitr, rmarkdown, testthat (>= 2.1.0), golem, DT, thematic
(>= 0.1.2)
Encoding: UTF-8
RoxygenNote: 7.2.3
VignetteBuilder: knitr
Collate: 'aaa_reimports.R' 'feedbacks.R' 'useful-items.R' 'tabs.R'
.....
RdMacros: lifecycle
Remotes: yogat3ch/UU
NeedsCompilation: no
Packaged: 2024-05-23 14:58:06 UTC; root
Author: David Granjon [aut, cre], RinteRface [cph], Almasaeed Studio
[ctb, cph] (AdminLTE3 theme for Bootstrap 4), Winston Chang
[ctb, cph] (Utils functions from shinydashboard)
Built: R 4.3.0; ; 2024-05-23 14:58:07 UTC; unix
RemoteType: github
RemoteHost: api.github.com
RemoteRepo: bs4Dash
RemoteUsername: yogat3ch
RemotePkgRef: yogat3ch/bs4Dash@HEAD
RemoteRef: HEAD
RemoteSha: 1d4d4618e47457067fd07e17c840e58f0213c684
GithubRepo: bs4Dash
GithubUsername: yogat3ch
GithubRef: HEAD
GithubSHA1: 1d4d4618e47457067fd07e17c840e58f0213c684
-- File: /usr/local/lib/R/site-library/bs4Dash/Meta/package.rds As part of the Dockerfile build process, one step is renv::restore which produces the following output:
RUN R --quiet -e "renv::restore()"
#10 [stage-4 5/5] RUN R --quiet -e "renv::restore()"
#10 0.368 > renv::restore()
#10 4.992
#10 6.615 ✔ Updated metadata database: 1.37 MB in 1 file.
#10 6.615
#10 6.616 ℹ Updating metadata database
#10 8.851 ✔ Updating metadata database ... done
#10 8.852
#10 9.077
#10 9.078
#10 9.240 ℹ No downloads are needed
#10 9.273 ✔ 2 pkgs + 1 dep: [5.3s]
#10 9.296 >
#10 9.296 >
#10 DONE 9.8sdespite the fact that in the renv.lock has the following entry (viewed from the output of renv::lockfile_read):
> rl$Packages$bs4Dash
$Package
[1] "bs4Dash"
$Version
[1] "2.3.3"
$Source
[1] "GitHub"
$Remotes
[1] "yogat3ch/UU"
$RemoteType
[1] "github"
$RemoteHost
[1] "api.github.com"
$RemoteRepo
[1] "bs4Dash"
$RemoteUsername
[1] "yogat3ch"
$RemoteRef
[1] "bc1e40f"
$RemoteSha
[1] "bc1e40f404915e566c0a70acc543288a0928989f"
$Requirements
[1] "UU" "bslib" "fresh" "grDevices" "htmltools" "httpuv"
[7] "httr" "jsonlite" "lifecycle" "shiny" "waiter"
$Hash
[1] "a707c272226b7265b5d2221f66a604c8"There's a discrepancy in version number and hash number that renv::restore doesn't seem to pick up on.
I've replicated this same behavior outside the Docker container on my Mac environment.
This is the output of .libPaths for context:
.libPaths()
[1] "/Users/stephenholsenbeck/Library/Caches/org.R-project.R/R/renv/library/dmdu-4393acb2/R-4.3/aarch64-apple-darwin20"
[2] "/Users/stephenholsenbeck/Library/Caches/org.R-project.R/R/renv/sandbox/R-4.3/aarch64-apple-darwin20/ac5c2659"Am I missing something critical about how renv::restore is supposed to work? I thought it was meant to synchronize the version of packages in the library with the versions of packages specified in the lockfile? I feel like renv::restore has synchronized based on the lockfile properly in the past when there was a version discrepancy, but it doesn't seem to be doing it anymore? Or is there some setting that's making it overlook the discrepancy in package versions?
Hi @kevinushey and renv dev team, We're using
renv::restoreas part of our Docker build process and just discovered that whenrenv::restoreruns while building the Docker image it's not respecting the RemoteSHA recorded in the lockfile for github dependencies, and is rather using the most recent commit. I was under the assumption thatrenv::restoreuses the RemoteSHA specified in the renv.lock file, but that doesn't seem to be the case. Am I missing a configuration option to ensure thatrenv::restoreuses the RemoteSHA from the lock file?