renv restore failing with Matrix

[devinrkeane]

Hi! renv::restore() is no longer working in any of our projects because Matrix gets in the way. I can install Matrix when running renv::install('Matrix@<version>') or renv::restore(packages='Matrix'), but running renv::restore() and I get inexplicably:

Is it related to this?

Matrix is available under R 4.5?

I cant seem to get past this.

Environment:

# Session Info ---------------------------------------------------------------
R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.1.2 tools_4.1.2    renv_1.0.7    

repos used:

c(CRAN_MC = "https://jfrog.ourcompanyrepo.io/artifactory/cran", RSPM = "https://packagemanager.posit.co/all/__linux__/jammy/latest", 
CRAN = "https://cran.r-project.org")

I've tried reordering every combination, i know this affects installs for our internal repo with artifactory.

Debugging through renv gets me to renv_available_packages_latest and the records are NULL:

I've tried available.packages() and Matrix is nowhere to be found.

I've tried libcurl, curl as download method, download.headers options.

Not sure how to attack this. Any help appreciated!

[kevinushey]

Can you please share the lockfile you're using, so I can try to reproduce locally?

[devinrkeane]

I have internal packages which will prevent shaing that one, however all you need to do is start a new project with renv, then add a package that imports Matrix (I tried MatrixModels), try to snapshot that with an install and I get the error.

This happens on my mac as well, not just ubuntu.

[kevinushey]

This doesn't happen for me on macOS. Using R 4.3.3:

> renv::snapshot()
The following required packages are not installed:
- MatrixModels
Packages must first be installed before renv can snapshot them.
Use `renv::dependencies()` to see where this package is used in your project.

What do you want to do? 

1: Snapshot, just using the currently installed packages.
2: Install the packages, then snapshot.
3: Cancel, and resolve the situation on your own.

Selection: 2
# Downloading packages -------------------------------------------------------
- Downloading MatrixModels from CRAN ...        OK [406.1 Kb in 0.51s]
Successfully downloaded 1 package in 2.2 seconds.

The following package(s) will be installed:
- MatrixModels [0.5-3]
These packages will be installed into "~/scratch/glumbu/renv/library/R-4.3/aarch64-apple-darwin20".

# Installing packages --------------------------------------------------------
- Installing MatrixModels ...                   OK [installed binary and cached in 0.46s]
Successfully installed 1 package in 0.49 seconds.
The following package(s) will be updated in the lockfile:

# CRAN -----------------------------------------------------------------------
- lattice        [* -> 0.22-5]
- Matrix         [* -> 1.6-5]
- MatrixModels   [* -> 0.5-3]

Do you want to proceed? [Y/n]: 

I see the same think on an Ubuntu 24.04 machine, using R 4.3.3. My suspicion is you're running into trouble because you're building R without recommended packages, so no installation of Matrix is available? For example, on my Linux VM, I have:

> find.package("Matrix")
[1] "/opt/R/4.3.3/lib/R/library/Matrix"
> packageVersion("Matrix")
[1] '1.6.5'

I suspect that, unless you're going to start building R with recommended packages available, you're going to have to manually install a compatible version of Matrix with e.g.

renv::install("Matrix@1.6-5")

[devinrkeane]

thanks, Sorry if this is getting out of renv's scope, really appreciate the help. I should note we're using kubeflow notebooks to run things but idk if that makes a difference since it's just a docker container.

It is true that the container did not install r-recommended. But we cannot install that right either, at least i dont know how - trying to lock versions of R in the docker build but it keeps installing r-base-core 4.4, this was not happening a month or so ago (dockerfile included).

minimal Dockerfile attached, if you run docker build --no-cache --tag=rtest -f reprex-Dockerfile.txt . You can see that r-base-core installs regardless of the hard-coded version (r-base).

reprex-Dockerfile.txt

Also working on trying through the rbuilds option but juggling lots of things so that's slower go.

May just need to capitulate and bump our version of R, 4.1 is seeming to be more trouble since this security patch.

[a-torgovitsky]

I think I'm also seeing this problem. Here's my output and I've attached my lockfile.

r$> sessionInfo()
R version 4.0.5 (2021-03-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Linux Mint 20

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base

loaded via a namespace (and not attached):
[1] compiler_4.0.5 tools_4.0.5    renv_1.0.7

r$> renv::restore()
- The library is already synchronized with the lockfile.

r$> renv::update()
- Checking for updated packages ... - GitHub authentication credentials are not available.
- Please set GITHUB_PAT, or ensure the 'gitcreds' package is installed.
- See https://usethis.r-lib.org/articles/git-credentials.html for more details.
Done!
The following package(s) will be updated:

# CRAN -----------------------------------------------------------------------
- AER             [1.2-9 -> 1.2-12]
- backports       [1.2.1 -> 1.5.0]
- bit             [4.0.4 -> 4.0.5]
- boot            [1.3-27 -> 1.3-30]
- bslib           [0.7.0 -> 0.8.0]
- car             [3.0-11 -> 3.1-2]
- carData         [3.0-4 -> 3.0-5]
- class           [7.3-18 -> 7.3-22]
- clipr           [0.7.1 -> 0.8.0]
- cluster         [2.1.1 -> 2.1.6]
- codetools       [0.2-18 -> 0.2-20]
- colorspace      [2.0-2 -> 2.1-1]
- conquer         [1.0.2 -> 1.3.3]
- cpp11           [0.4.3 -> 0.4.7]
- crayon          [1.4.1 -> 1.5.3]
- data.table      [1.14.0 -> 1.15.4]
- diffobj         [0.3.4 -> 0.3.5]
- fansi           [0.5.0 -> 1.0.6]
- farver          [2.1.0 -> 2.1.2]
- foreach         [1.5.1 -> 1.5.2]
- foreign         [0.8-84 -> 0.8-87]
- gert            [2.1.0 -> 2.1.1]
- glue            [1.6.2 -> 1.7.0]
- gtable          [0.3.0 -> 0.3.5]
- hms             [1.1.0 -> 1.1.3]
- isoband         [0.2.5 -> 0.2.7]
- iterators       [1.0.13 -> 1.0.14]
- KernSmooth      [2.23-18 -> 2.23-24]
- labeling        [0.4.2 -> 0.4.3]
- lattice         [0.20-41 -> 0.22-6]
- lifecycle       [1.0.3 -> 1.0.4]
- lme4            [1.1-27.1 -> 1.1-35.5]
- markdown        [1.1 -> 1.13]
- MatrixModels    [0.5-0 -> 0.5-3]
- matrixStats     [0.59.0 -> 1.3.0]
- mgcv            [1.8-34 -> 1.9-1]
- mime            [0.11 -> 0.12]
- minqa           [1.2.4 -> 1.2.8]
- munsell         [0.5.0 -> 0.5.1]
- nlme            [3.1-152 -> 3.1-166]
- nloptr          [1.2.2.2 -> 2.1.1]
- openssl         [2.2.0 -> 2.2.1]
- openxlsx        [4.2.5.2 -> 4.2.6.1]
- prettyunits     [1.1.1 -> 1.2.0]
- progress        [1.2.2 -> 1.2.3]
- quantreg        [5.86 -> 5.98]
- RColorBrewer    [1.1-2 -> 1.1-3]
- Rcpp            [1.0.7 -> 1.0.13]
- RcppArmadillo   [0.10.6.0.0 -> 14.0.0-1]
- RcppEigen       [0.3.3.9.1 -> 0.3.4.0.1]
- readr           [2.0.0 -> 2.1.5]
- readxl          [1.3.1 -> 1.4.3]
- rematch         [1.0.1 -> 2.0.0]
- rio             [0.5.27 -> 1.2.1]
- rmarkdown       [2.27 -> 2.28]
- rpart           [4.1-15 -> 4.1.23]
- rprojroot       [2.0.2 -> 2.0.4]
- rstudioapi      [0.13 -> 0.16.0]
- shape           [1.4.6 -> 1.4.6.1]
- shiny           [1.8.1.1 -> 1.9.1]
- sp              [1.4-5 -> 2.1-4]
- SparseM         [1.81 -> 1.84-2]
- spatial         [7.3-13 -> 7.3-17]
- stringi         [1.7.3 -> 1.8.4]
- survival        [3.2-10 -> 3.7-0]
- tidyselect      [1.2.0 -> 1.2.1]
- tzdb            [0.3.0 -> 0.4.0]
- usethis         [2.2.3 -> 3.0.0]
- utf8            [1.2.1 -> 1.2.4]
- viridisLite     [0.4.0 -> 0.4.2]
- vroom           [1.5.3 -> 1.6.5]
- withr           [2.5.0 -> 3.0.1]
- xfun            [0.46 -> 0.47]
- xgboost         [1.7.7.1 -> 1.7.8.1]
- yaml            [2.2.1 -> 2.3.10]
- zip             [2.2.0 -> 2.3.1]
- zoo             [1.8-9 -> 1.8-12]

Do you want to proceed? [Y/n]: y

# Downloading packages -------------------------------------------------------
- Downloading AER from CRAN ...                 OK [file is up to date]
- Downloading car from CRAN ...                 OK [file is up to date]
- Downloading mgcv from CRAN ...                OK [file is up to date]
- Downloading lattice from CRAN ...             OK [file is up to date]
- Downloading nlme from CRAN ...                OK [file is up to date]
- Downloading lme4 from CRAN ...                OK [file is up to date]
- Downloading boot from CRAN ...                OK [file is up to date]
- Downloading minqa from CRAN ...               OK [file is up to date]
- Downloading Rcpp from CRAN ...                OK [file is up to date]
- Downloading nloptr from CRAN ...              OK [file is up to date]
- Downloading RcppEigen from CRAN ...           OK [file is up to date]
- Downloading backports from CRAN ...           OK [file is up to date]
- Downloading glue from CRAN ...                OK [file is up to date]
- Downloading lifecycle from CRAN ...           OK [file is up to date]
- Downloading fansi from CRAN ...               OK [file is up to date]
- Downloading utf8 from CRAN ...                OK [file is up to date]
- Downloading tidyselect from CRAN ...          OK [file is up to date]
- Downloading withr from CRAN ...               OK [file is up to date]
- Downloading stringi from CRAN ...             OK [file is up to date]
- Downloading cpp11 from CRAN ...               OK [file is up to date]
- Downloading xfun from CRAN ...                OK [file is up to date]
- Downloading yaml from CRAN ...                OK [file is up to date]
- Downloading quantreg from CRAN ...            OK [file is up to date]
- Downloading MatrixModels from CRAN ...        OK [file is up to date]
Error: package 'Matrix' is not available
In addition: Warning messages:
1: GITHUB_PAT is unset. Updates may fail due to GitHub's API rate limit.

To increase your GitHub API rate limit:
- Use `usethis::create_github_token()` to create a Personal Access Token (PAT).
- Use `usethis::edit_r_environ()` and add the token as `GITHUB_PAT`.
2: In value[[3L]](cond) : Could not find any credentials
Traceback (most recent calls last):
52: renv::update()
51: install(packages = updates, library = libpaths, rebuild = rebuild,
        prompt = prompt, lock = lock, project = project)
50: retrieve(packages)
49: handler(package, renv_retrieve_impl(package))
48: renv_retrieve_impl(package)
47: renv_retrieve_repos(record)
46: renv_retrieve_repos_impl(record)
45: renv_retrieve_package(record, url, path)
44: renv_retrieve_successful(record, path)
43: (function() {
        repos <- if (is.null(desc$biocViews))
            getOption("repos")
        else renv_bioconductor_repos()
        renv_scope_options(repos = repos)
        renv_retrieve_successful_recurse(deps)
    })()
42: renv_retrieve_successful_recurse(deps)
41: renv_retrieve_successful_recurse_impl(remote)
40: dynamic(key = list(remote = remote), value = renv_retrieve_successful_recurse_impl_one(remote),
        force = force)
39: the$dynamic_objects[[id]] %||% {
        dlog("dynamic", "memoizing dynamic value for '%s'", id)
        value
    }
38: renv_retrieve_successful_recurse_impl_one(remote)
37: renv_retrieve_impl(remote)
36: renv_retrieve_repos(record)
35: renv_retrieve_repos_impl(record)
34: renv_retrieve_package(record, url, path)
33: renv_retrieve_successful(record, path)
32: (function() {
        repos <- if (is.null(desc$biocViews))
            getOption("repos")
        else renv_bioconductor_repos()
        renv_scope_options(repos = repos)
        renv_retrieve_successful_recurse(deps)
    })()
31: renv_retrieve_successful_recurse(deps)
30: renv_retrieve_successful_recurse_impl(remote)
29: dynamic(key = list(remote = remote), value = renv_retrieve_successful_recurse_impl_one(remote),
        force = force)
28: the$dynamic_objects[[id]] %||% {
        dlog("dynamic", "memoizing dynamic value for '%s'", id)
        value
    }
27: renv_retrieve_successful_recurse_impl_one(remote)
26: renv_retrieve_impl(remote)
25: renv_retrieve_repos(record)
24: renv_retrieve_repos_impl(record)
23: renv_retrieve_package(record, url, path)
22: renv_retrieve_successful(record, path)
21: (function() {
        repos <- if (is.null(desc$biocViews))
            getOption("repos")
        else renv_bioconductor_repos()
        renv_scope_options(repos = repos)
        renv_retrieve_successful_recurse(deps)
    })()
20: renv_retrieve_successful_recurse(deps)
19: renv_retrieve_successful_recurse_impl(remote)
18: dynamic(key = list(remote = remote), value = renv_retrieve_successful_recurse_impl_one(remote),
        force = force)
17: the$dynamic_objects[[id]] %||% {
        dlog("dynamic", "memoizing dynamic value for '%s'", id)
        value
    }
16: renv_retrieve_successful_recurse_impl_one(remote)
15: renv_retrieve_impl(remote)
14: renv_retrieve_repos(record)
13: renv_retrieve_repos_impl(record)
12: renv_retrieve_package(record, url, path)
11: renv_retrieve_successful(record, path)
10: (function() {
        repos <- if (is.null(desc$biocViews))
            getOption("repos")
        else renv_bioconductor_repos()
        renv_scope_options(repos = repos)
        renv_retrieve_successful_recurse(deps)
    })()
 9: renv_retrieve_successful_recurse(deps)
 8: renv_retrieve_successful_recurse_impl(remote)
 7: dynamic(key = list(remote = remote), value = renv_retrieve_successful_recurse_impl_one(remote),
        force = force)
 6: the$dynamic_objects[[id]] %||% {
        dlog("dynamic", "memoizing dynamic value for '%s'", id)
        value
    }
 5: renv_retrieve_successful_recurse_impl_one(remote)
 4: renv_retrieve_impl(remote)
 3: renv_available_packages_latest(package)
 2: stopf("package '%s' is not available", package)
 1: stop(sprintf(fmt, ...), call. = call.)
r$>

renv.lock.txt

[a-torgovitsky]

Upgrading R to 4.4.1 fixed my issues