Renv 1.0.0 broke Gitlab CI pipeline

lschneiderbauer commented 1 year ago

Hi,

I am using renv with Gitlab-CI (setup as described in your vignette). The update to version 1.0.0 broke the pipeline:

Here the (imo) relevant lines from the CI-Log:

$ cd $CI_PROJECT_DIR
$ Rscript -e "if (!requireNamespace('renv', quietly = TRUE)) install.packages('renv')"
# Bootstrapping renv 1.0.0 ---------------------------------------------------
- Downloading renv ... 
OK
- Installing renv  ... 
OK
Error: Error installing package 'mockery':
===================================
ERROR: dependency ‘testthat’ is not available for package ‘mockery’
* removing ‘/builds/lschneiderbauer/scmqueries/renv/staging/1/mockery’
install of package 'mockery' failed [error code 1]
Execution halted

Renv 0.17 used to work fine. The used docker image is: rocker/r-base:4.2.3

I cleared the runner cache, but the issue remains. Any clue what could be wrong here?

kevinushey commented 1 year ago

Can you share the whole log? The fact that installation is failing when trying to install the mockery package makes me think this is a problem on the R package repository side, but it's hard to know for sure without more information.

lschneiderbauer commented 1 year ago

Sure, this is the whole log:

gitlab_devops_raw.txt

Thanks for you help!

kevinushey commented 1 year ago

We also just released renv 1.0.1 to CRAN; do you see the issue with that version as well?

lschneiderbauer commented 1 year ago

Hi,

I am seeing a different but very similar issue: gitlab_devops log renv 1.0.1

This time it complains that 'cli' and 'R6' are not available for package 'desc'. However, the log above seems to indicate that they downloaded fine.

kevinushey commented 1 year ago

Thanks! Could you please share the lockfile you're using here? It looks like renv is erroneously attempting to install desc before cli, even though desc depends on cli.

kevinushey commented 1 year ago

It's also very odd that cli appears in the download log twice.

lschneiderbauer commented 1 year ago

renv.lock.txt

This is the renv.lock file used above. I renamed it to ".txt" because Github didn't like it otherwise. I didn't touch it manually, it was created by renv::snapshot(). (with the "explicit" option).

It's quite strange. desc does not even appear as a requirement for cli. Edit: Nevermind, I am confused, it should be the other way arround I guess? But desc does not appear at all?

lschneiderbauer commented 1 year ago

In case you would want to know how the "DESCRIPTION" file looks like:

Depends:
    dplyr (>= 1.1.0),
    dbplyr (>= 2.3.1),
    R (>= 4.1)
Imports:
    DBI (>= 1.1.1),
    lubridate (>= 1.7.10),
    tidyr (>= 1.2.0),
    tibble (>= 3.1.8),
    stringr (>= 1.4.0),
    odbc (>= 1.3.5),
    rlang (>= 1.0.6),
    purrr (>= 0.3.4),
    crayon (>= 1.4.1),
    glue (>= 1.4.2),
    keyring (>= 1.2.0),
    rstudioapi (>= 0.14),
    lifecycle,
    cli (>= 3.0.0),
    prettyunits (>= 1.1.1),
    duckdb (>= 0.8.0),
    duckdb (< 0.9.0)
Suggests: 
    devtools,
    testthat (>= 3.0.0),
    knitr,
    rmarkdown,
    styler,
    lintr,
    DT (>= 0.28)
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.2.3
Config/testthat/edition: 3

lschneiderbauer commented 1 year ago

It's also very odd that cli appears in the download log twice.

I only see it once actually, in line 2188. Did you mistake clipr for cli ?

kevinushey commented 1 year ago

I only see it once actually, in line 2188. Did you mistake clipr for cli ?

😮‍💨 You are right; sorry, my lack of sleep is catching up to me today... Either way, I can take a closer look.

kevinushey commented 1 year ago

Unfortunately, it does seem to work on my machine -- although I had to switch the repositories in the lockfile to instead use the public PPM instance.

> renv::restore(rebuild = TRUE)
The following package(s) will be updated:

# RSPM -----------------------------------------------------------------------
- askpass       [* -> 1.1]
- assertthat    [* -> 0.2.1]
- bit           [* -> 4.0.5]
- bit64         [* -> 4.0.5]
- blob          [* -> 1.2.4]
- cli           [* -> 3.6.1]
- cpp11         [* -> 0.4.6]
- crayon        [* -> 1.5.2]
- DBI           [* -> 1.1.3]
- dbplyr        [* -> 2.3.3]
- dplyr         [* -> 1.1.2]
- duckdb        [* -> 0.8.1-1]
- fansi         [* -> 1.0.4]
- filelock      [* -> 1.0.2]
- generics      [* -> 0.1.3]
- glue          [* -> 1.6.2]
- hms           [* -> 1.1.3]
- keyring       [* -> 1.3.1]
- lifecycle     [* -> 1.0.3]
- lubridate     [* -> 1.9.2]
- magrittr      [* -> 2.0.3]
- odbc          [* -> 1.3.5]
- openssl       [* -> 2.1.0]
- pillar        [* -> 1.9.0]
- pkgconfig     [* -> 2.0.3]
- prettyunits   [* -> 1.1.1]
- purrr         [* -> 1.0.2]
- R6            [* -> 2.5.1]
- rappdirs      [* -> 0.3.3]
- Rcpp          [* -> 1.0.11]
- rlang         [* -> 1.1.1]
- rstudioapi    [* -> 0.15.0]
- sodium        [* -> 1.2.1]
- stringi       [* -> 1.7.12]
- stringr       [* -> 1.5.0]
- sys           [* -> 3.4.2]
- tibble        [* -> 3.2.1]
- tidyr         [* -> 1.3.0]
- tidyselect    [* -> 1.2.0]
- timechange    [* -> 0.2.0]
- utf8          [* -> 1.2.3]
- vctrs         [* -> 0.6.3]
- withr         [* -> 2.5.0]
- yaml          [* -> 2.3.7]

Do you want to proceed? [Y/n]: y

# Downloading packages -------------------------------------------------------
- Downloading DBI from RSPM ...                 OK [712.7 Kb in 0.99s]
- Downloading R6 from RSPM ...                  OK [62.6 Kb in 0.46s]
- Downloading Rcpp from RSPM ...                OK [2.9 Mb in 0.94s]
- Downloading askpass from RSPM ...             OK [5.6 Kb in 0.86s]
- Downloading sys from RSPM ...                 OK [19.8 Kb in 0.84s]
- Downloading assertthat from RSPM ...          OK [file is up to date]
- Downloading bit from RSPM ...                 OK [809.3 Kb in 1.0s]
- Downloading bit64 from RSPM ...               OK [132.7 Kb in 0.74s]
- Downloading blob from RSPM ...                OK [10.4 Kb in 0.72s]
- Downloading rlang from RSPM ...               OK [file is up to date]
- Downloading vctrs from RSPM ...               OK [944 Kb in 0.46s]
- Downloading cli from RSPM ...                 OK [552.1 Kb in 0.61s]
- Downloading glue from RSPM ...                OK [105.1 Kb in 0.67s]
- Downloading lifecycle from RSPM ...           OK [104.9 Kb in 0.58s]
- Downloading cpp11 from RSPM ...               OK [file is up to date]
- Downloading crayon from RSPM ...              OK [39.8 Kb in 1.2s]
- Downloading dbplyr from RSPM ...              OK [703.4 Kb in 0.76s]
- Downloading dplyr from RSPM ...               OK [1 Mb in 0.51s]
- Downloading generics from RSPM ...            OK [169.6 Kb in 0.4s]
- Downloading magrittr from RSPM ...            OK [260.9 Kb in 0.45s]
- Downloading pillar from RSPM ...              OK [431.6 Kb in 0.41s]
- Downloading fansi from RSPM ...               OK [470.5 Kb in 0.44s]
- Downloading utf8 from RSPM ...                OK [234.4 Kb in 0.47s]
- Downloading tibble from RSPM ...              OK [552.5 Kb in 0.39s]
- Downloading pkgconfig from RSPM ...           OK [6 Kb in 0.38s]
- Downloading tidyselect from RSPM ...          OK [99.1 Kb in 0.4s]
- Downloading withr from RSPM ...               OK [100.5 Kb in 0.42s]
- Downloading purrr from RSPM ...               OK [file is up to date]
- Downloading tidyr from RSPM ...               OK [799.6 Kb in 0.48s]
- Downloading stringr from RSPM ...             OK [171.8 Kb in 0.43s]
- Downloading stringi from RSPM ...             OK [7.3 Mb in 0.62s]
- Downloading duckdb from RSPM ...              OK [file is up to date]
- Downloading filelock from RSPM ...            OK [file is up to date]
- Downloading hms from RSPM ...                 OK [43.4 Kb in 0.69s]
- Downloading keyring from RSPM ...             OK [file is up to date]
- Downloading openssl from RSPM ...             OK [file is up to date]
- Downloading sodium from RSPM ...              OK [file is up to date]
- Downloading yaml from RSPM ...                OK [92.4 Kb in 0.7s]
- Downloading rappdirs from RSPM ...            OK [12 Kb in 0.73s]
- Downloading lubridate from RSPM ...           OK [417.6 Kb in 0.42s]
- Downloading timechange from RSPM ...          OK [100.6 Kb in 0.39s]
- Downloading odbc from RSPM ...                OK [file is up to date]
- Downloading prettyunits from RSPM ...         OK [10.1 Kb in 0.72s]
- Downloading rstudioapi from RSPM ...          OK [112 Kb in 0.7s]
Successfully downloaded 44 packages in 41 seconds.

# Installing packages --------------------------------------------------------
- Installing DBI ...                            OK [built from source and cached in 3.3s]
- Installing R6 ...                             OK [built from source and cached in 1.6s]
- Installing Rcpp ...                           OK [built from source and cached in 16s]
- Installing sys ...                            OK [built from source and cached in 2.3s]
- Installing askpass ...                        OK [built from source and cached in 2.1s]
- Installing assertthat ...                     OK [built from source and cached in 1.5s]
- Installing bit ...                            OK [built from source and cached in 5.5s]
- Installing bit64 ...                          OK [built from source and cached in 5.4s]
- Installing rlang ...                          OK [built from source and cached in 6.7s]
- Installing cli ...                            OK [built from source and cached in 6.1s]
- Installing glue ...                           OK [built from source and cached in 1.7s]
- Installing lifecycle ...                      OK [built from source and cached in 1.4s]
- Installing vctrs ...                          OK [built from source and cached in 11s]
- Installing blob ...                           OK [built from source and cached in 1.4s]
- Installing cpp11 ...                          OK [built from source and cached in 1.2s]
- Installing crayon ...                         OK [built from source and cached in 1.5s]
- Installing generics ...                       OK [built from source and cached in 1.2s]
- Installing magrittr ...                       OK [built from source and cached in 1.7s]
- Installing fansi ...                          OK [built from source and cached in 3.3s]
- Installing utf8 ...                           OK [built from source and cached in 3.4s]
- Installing pillar ...                         OK [built from source and cached in 2.6s]
- Installing pkgconfig ...                      OK [built from source and cached in 0.99s]
- Installing tibble ...                         OK [built from source and cached in 3.0s]
- Installing withr ...                          OK [built from source and cached in 1.3s]
- Installing tidyselect ...                     OK [built from source and cached in 1.7s]
- Installing dplyr ...                          OK [built from source and cached in 6.5s]
- Installing purrr ...                          OK [built from source and cached in 4.7s]
- Installing stringi ...                        OK [built from source and cached in 3.6m]
- Installing stringr ...                        OK [built from source and cached in 4.4s]
- Installing tidyr ...                          OK [built from source and cached in 12s]
- Installing dbplyr ...                         OK [built from source and cached in 9.2s]
- Installing duckdb ...                         OK [built from source and cached in 7.3m]
- Installing filelock ...                       OK [built from source and cached in 1.7s]
- Installing hms ...                            OK [built from source and cached in 1.5s]
- Installing openssl ...                        OK [built from source and cached in 6.4s]
- Installing sodium ...                         OK [built from source and cached in 4.4s]
- Installing yaml ...                           OK [built from source and cached in 3.3s]
- Installing rappdirs ...                       OK [built from source and cached in 1.6s]
- Installing keyring ...                        OK [built from source and cached in 4.8s]
- Installing timechange ...                     OK [built from source and cached in 7.8s]
- Installing lubridate ...                      OK [built from source and cached in 4.3s]
- Installing odbc ...                           OK [built from source and cached in 20s]
- Installing prettyunits ...                    OK [built from source and cached in 1.1s]
- Installing rstudioapi ...                     OK [built from source and cached in 1.4s]

Looking a bit more at your log, there seems to be a strange disconnect between the packages that renv wants to restore, and the packages that are actually getting downloaded and installed. I'm not sure what to make of that -- do you have any ideas? (For example, why is the brio package being downloaded here?)

It seems like renv::restore() is also attempting to install some of the packages recorded in the DESCRIPTION file, but I'm not sure why that would be the case here.

lschneiderbauer commented 1 year ago

It seems like renv::restore() is also attempting to install some of the packages recorded in the DESCRIPTION file, but I'm not sure why that would be the case here.

In addition, there seems to be the pattern that all the "correct" packages that are supposed to be downloaded are from "RSPM", and all the others from "CRAN". I am not sure what to make of that.

Maybe worth mentioning: I set the options(repos = c("CRAN"="some-url", "dev"="some-url") in the .RProfile file to our internal repositories. But I thought that that should have no influence on renv, since I use the override mechanism described in the CI vignette: RENV_CONFIG_REPOS_OVERRIDE: "http://cran.r-project.org". Our Gitlab-CI cannot access the internal repository, but it works with the "outside"-ones. Since the packages downloaded fine, I assumed that this works as it is configured now.

kevinushey commented 1 year ago

Hmm, I wonder if it's related to this:

https://github.com/rstudio/renv/blob/250ea841644012eeda07f5226605037e148d938f/inst/resources/activate.R#L159-L170

I wonder if the issue reproduces if one is using both PPM and CRAN repositories at the same time?

kevinushey commented 1 year ago

I guess the other thing worth asking is: are you able to reproduce the issue locally? If so, any chance you can share a reproducible example (e.g. a GitHub project that I could try to clone and renv::restore() to reproduce myself)?

lschneiderbauer commented 1 year ago

Regarding a local reproduction:

If I remove the renv/library and renv/staging folder and restart the session, I indeed run into the same problem that renv tries to download packages which have nothing to do the renv.lock file. (However, the actual error message is different: It now complains that it cannot build the sys package. That it cannot build the sys package is expected (proper build tools are missing), but why it decides to not take the binary package is unclear to me.)
I still do not understand this mixture between "RSPM" and "CRAN" repositories in the download-log. All of the packages should be available in the company-intern R repository (options("CRAN") points to the internal URL). So it should either always say "RSPM" or always say "CRAN", no? This mix-up also happens locally.
Unfortunately I cannot just share the full package since it's internal. But I will try to find a minimal reproducible example which can be shared. It seems somehow that our internal repositories could have something to do with the problem? In this case it is probably unlikely to get something reproducible outside of our internal environment.

lschneiderbauer commented 1 year ago

Okay, I think I discovered the main reason it was downloading DESCRIPTION-unrelated packages. I had

  "package.dependency.fields": [
    "Imports",
    "Depends",
    "LinkingTo",
    "Suggests"
  ],

in renv/settings.json. I originally put that in because I hoped that the "explicit" snapshot would pick up the "Suggests" part of the DESCRIPTION file as well (I think I know now that it does not). If I remove the "Suggests" part, restoration works again.

rstudio / renv

Renv 1.0.0 broke Gitlab CI pipeline #1632