Error in `rec$remote[[1]]`: subscript out of bounds

stephenashton-dhsc commented 1 month ago

Hi,

I'm trying to migrate to using pak for package installation/management on a Linux system and I'm getting a strange error whenever I try to install a package (the example below is for a CRAN package, but I get the same messages for a local package).

For example, if I try to run:

pak::pak("ggplot2")

I get the following error message:

Error:  1  deps for  0/1  pkgs [⠋] Resolving standard (CRAN/BioC) packages
! error in pak subprocess
Caused by error in `rec$remote[[1]]`:
! subscript out of bounds
Type .Last.error to see the more details.

If I call .Last.error, I get the following details:

<callr_error/rlib_error_3_0/rlib_error/error>
Error:
! error in pak subprocess
Caused by error in `rec$remote[[1]]`:
! subscript out of bounds
---
Backtrace:
1. pak::pak("ggplot2")
2. pak::pkg_install(pkg, ...)
3. pak:::remote(function(...) get("pkg_install_make_plan", asNamespace("pak"))(...), …
4. err$throw(res$error)
---
Subprocess backtrace:
1. base::withCallingHandlers(cli_message = function(msg) { …
2. get("pkg_install_make_plan", asNamespace("pak"))(...)
3. prop$solve()
4. private$plan$solve(policy = private$policy)
5. pkgdepends:::pkgplan_solve(self, private, match.arg(policy))
6. self$resolve()
7. pkgdepends:::pkgplan_resolve(self, private)
8. pkgdepends:::synchronise(self$async_resolve())
9. asNamespace("pkgcache")$synchronise(...)
10. base::stop(priv$value)
11. global (function (e) …

This is under the following setup:

> getOption("repos")
                         CRAN
"https://cloud.r-project.org"
> pak::pak_sitrep()
* pak version:
- 0.7.2
* Version information:
- pak platform: x86_64-pc-linux-gnu (current: x86_64-pc-linux-gnu, compatible)
- pak repository: - (local install?)
* Optional packages installed:
- pillar
* Library path:
- /data/users/hemt/R-library
- /usr/local/packages/R/4.3.2/lib64/R/library
* pak is installed at /data/users/hemt/R-library/pak.
* Dependency versions:
- callr      3.7.5
- cli        3.6.2
- curl       5.2.1
- desc       1.4.3
- filelock   1.0.3
- jsonlite   1.8.8
- lpSolve    5.6.20
- pkgbuild   1.4.4
- pkgcache   2.2.1
- pkgdepends 0.7.2
- pkgsearch  3.1.3
- processx   3.8.4
- ps         1.7.6
- R6         2.5.1
- zip        2.3.1
* Dependencies can be loaded

Any advice/support you could provide on the above would be greatly appreciated!

gaborcsardi commented 1 month ago

How did you install pak?

stephenashton-dhsc commented 1 month ago

Via install.packages("pak")

gaborcsardi commented 1 month ago

Can you try installing it as shown at https://pak.r-lib.org/reference/install.html#pre-built-binaries?

stephenashton-dhsc commented 1 month ago

Unfortunately, it doesn't appear as though that has changed anything:

> install.packages("pak", repos = sprintf(
+ "https://r-lib.github.io/p/pak/stable/%s/%s/%s",
+ .Platform$pkgType,
+ R.Version()$os,
+ R.Version()$arch
+ ))
Installing package into ‘/data/users/hemt/R-library’
(as ‘lib’ is unspecified)
trying URL 'https://r-lib.github.io/p/pak/stable/source/linux-gnu/x86_64/src/contrib/../../../../../linux/x86_64/pak_0.7.2_R-4-3_x86_64-linux.tar.gz'
Content type 'application/gzip' length 8038050 bytes (7.7 MB)
==================================================
downloaded 7.7 MB

* installing *binary* package ‘pak’ ...
* DONE (pak)

The downloaded source packages are in
        ‘/tmp/RtmpohPr1e/downloaded_packages’
> pak::pak("ggplot2")
Error: ing metadata database [0/8] | Downloading [0 B / 0 B]
! error in pak subprocess
Caused by error in `rec$remote[[1]]`:
! subscript out of bounds
Type .Last.error to see the more details.
> .Last.error
<callr_error/rlib_error_3_0/rlib_error/error>
Error:
! error in pak subprocess
Caused by error in `rec$remote[[1]]`:
! subscript out of bounds
---
Backtrace:
1. pak::pak("ggplot2")
2. pak::pkg_install(pkg, ...)
3. pak:::remote(function(...) get("pkg_install_make_plan", asNamespace("pak"))(...), …
4. err$throw(res$error)
---
Subprocess backtrace:
1. base::withCallingHandlers(cli_message = function(msg) { …
2. get("pkg_install_make_plan", asNamespace("pak"))(...)
3. prop$solve()
4. private$plan$solve(policy = private$policy)
5. pkgdepends:::pkgplan_solve(self, private, match.arg(policy))
6. self$resolve()
7. pkgdepends:::pkgplan_resolve(self, private)
8. pkgdepends:::synchronise(self$async_resolve())
9. asNamespace("pkgcache")$synchronise(...)
10. base::stop(priv$value)
11. global (function (e) …

gaborcsardi commented 1 month ago

This is definitely a bug in pak, and I think it is triggered by some metadata in the DESCRIPTION file of one of your installed packages. Can you please try to install the pkgdepends package and try this:

p <- pkgdepends::new_pkg_installation_proposal("ggplot2")
p$resolve()
p$solve()

This should error as well, but hopefully with a better stack trace. Thanks a lot!

stephenashton-dhsc commented 1 month ago

Unfortunately, it's giving less info:

> p <- pkgdepends::new_pkg_installation_proposal("ggplot2")
> p$resolve()
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
> p$solve()
Error in `pkgplan_solve(self, private, match.arg(policy))`:
! Package list has changed, you need to call the `$resolve()` method again?
Type .Last.error to see the more details.

gaborcsardi commented 1 month ago

After the

Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages

can you type traceback() and send the output?

stephenashton-dhsc commented 1 month ago

Ah, apologies:

> p$resolve()
Error in rec$remote[[1]] : subscript out of bounds
> traceback()
6: stop(priv$value)
5: asNamespace("pkgcache")$synchronise(...)
4: synchronise(self$async_resolve())
3: pkgplan_resolve(self, private)
2: private$plan$resolve()
1: p$resolve()

gaborcsardi commented 1 month ago

Sorry for the extra work, just one more thing. Can you please run this? It tries to parse the metadata already installed versions of the ggplot2 dependencies to see which one fails.

pkgs <- pak::pkg_deps("ggplot2")$package
for (pkg in pkgs) { 
  print(pkg) 
  pkgdepends::new_pkg_installation_proposal(
    pkg, 
    config = list(dependencies = FALSE)
  )$resolve() 
}

and then when it (hopefully) fails, send me the output of

packageDescription(pkg)

(pkg should be the package for which it failed.)

Thanks a lot again.

stephenashton-dhsc commented 1 month ago

> pkgs <- pak::pkg_deps("ggplot2")$package
✔ Updated metadata database: 2.64 MB in 2 files.
✔ Updating metadata database ... done
> for (pkg in pkgs) {
+ print(pkg)
+ pkgdepends::new_pkg_installation_proposal(
+ pkg,
+ config = list(dependencies = FALSE)
+ )$resolve()
+ }
[1] "cli"
✔ Loading metadata database ... done
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
> packageDescription(pkg)
Package: cli
Title: Helpers for Developing Command Line Interfaces
Version: 3.6.1
Authors@R: c( person("Gábor", "Csárdi", , "csardi.gabor@gmail.com",
        role = c("aut", "cre")), person("Hadley", "Wickham", role =
        "ctb"), person("Kirill", "Müller", role = "ctb"),
        person("RStudio", role = c("cph", "fnd")) )
Description: A suite of tools to build attractive command line
        interfaces ('CLIs'), from semantic elements: headings, lists,
        alerts, paragraphs, etc. Supports custom themes via a
        'CSS'-like language. It also contains a number of lower level
        'CLI' elements: rules, boxes, trees, and 'Unicode' symbols with
        'ASCII' alternatives. It support ANSI colors and text styles as
        well.
License: MIT + file LICENSE
URL: https://cli.r-lib.org/, https://github.com/r-lib/cli#readme
BugReports: https://github.com/r-lib/cli/issues
Depends: R (>= 3.4)
Imports: utils
Suggests: callr, covr, crayon, digest, glue (>= 1.6.0), grDevices,
        htmltools, htmlwidgets, knitr, methods, mockery, processx, ps
        (>= 1.3.4.9000), rlang (>= 1.0.2.9003), rmarkdown, rprojroot,
        rstudioapi, testthat, tibble, whoami, withr
Config/Needs/website: r-lib/asciicast, bench, brio, cpp11, decor, desc,
        fansi, prettyunits, sessioninfo, tidyverse/tidytemplate,
        usethis, vctrs
Config/testthat/edition: 3
Encoding: UTF-8
RoxygenNote: 7.2.1.9000
NeedsCompilation: yes
Packaged: 2023-03-22 13:59:32 UTC; gaborcsardi
Author: Gábor Csárdi [aut, cre], Hadley Wickham [ctb], Kirill Müller
        [ctb], RStudio [cph, fnd]
Maintainer: Gábor Csárdi <csardi.gabor@gmail.com>
Repository: CRAN
Date/Publication: 2023-03-23 12:52:05 UTC
Built: R 4.3.2; x86_64-pc-linux-gnu; 2023-11-20 15:39:32 UTC; unix

-- File: /usr/local/packages/R/4.3.2/lib64/R/library/cli/Meta/package.rds

stephenashton-dhsc commented 1 month ago

Just to note, I've tried stepping through manually and it doesn't appear to be a problem with cli specifically:

> i <- 2; print(pkgs[i]); pkgdepends::new_pkg_installation_proposal(pkgs[i], config = list(dependencies = FALSE))$resolve()
[1] "colorspace"
Error in rec$remote[[1]] : subscript out of bounds
> i <- 3; print(pkgs[i]); pkgdepends::new_pkg_installation_proposal(pkgs[i], config = list(dependencies = FALSE))$resolve()
[1] "fansi"
Error in rec$remote[[1]] : subscript out of boundsled packages
> i <- 4; print(pkgs[i]); pkgdepends::new_pkg_installation_proposal(pkgs[i], config = list(dependencies = FALSE))$resolve()
[1] "farver"
Error in rec$remote[[1]] : subscript out of bounds

stephenashton-dhsc commented 1 month ago

Actually, it appears to be a problem with all of the packages:

> for (pkg in pkgs) {
+ print(pkg)
+ try(pkgdepends::new_pkg_installation_proposal(pkg, config = list(dependencies = FALSE))$resolve())
+ }
[1] "cli"
Error in rec$remote[[1]] : subscript out of bounds
[1] "colorspace"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "fansi"
Error in rec$remote[[1]] : subscript out of bounds
[1] "farver"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "ggplot2"
Error in rec$remote[[1]] : subscript out of bounds
[1] "glue"
Error in rec$remote[[1]] : subscript out of bounds
[1] "gtable"
Error in rec$remote[[1]] : subscript out of bounds
[1] "isoband"
Error in rec$remote[[1]] : subscript out of bounds
[1] "labeling"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "lattice"
Error in rec$remote[[1]] : subscript out of bounds
[1] "lifecycle"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "magrittr"
Error in rec$remote[[1]] : subscript out of bounds
[1] "mgcv"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "munsell"
Error in rec$remote[[1]] : subscript out of bounds
[1] "nlme"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "pillar"
Error in rec$remote[[1]] : subscript out of bounds
[1] "pkgconfig"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "R6"
Error in rec$remote[[1]] : subscript out of bounds
[1] "RColorBrewer"
Error in rec$remote[[1]] : subscript out of bounds
[1] "rlang"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "scales"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "tibble"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "utf8"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "vctrs"
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
[1] "viridisLite"
Error in rec$remote[[1]] : subscript out of bounds
[1] "withr"
Error in rec$remote[[1]] : subscript out of bounds
[1] "MASS"
Error in rec$remote[[1]] : subscript out of bounds
[1] "Matrix"
Error in rec$remote[[1]] : subscript out of bounds

gaborcsardi commented 1 month ago

I think my test is just not working and your packages are fine.

gaborcsardi commented 1 month ago

I can try to fix this without knowing why it happens, but if you are up for it, you can debug it after installing the debug branch of pkgdepends:

pak::pkg_install("r-lib/pkgdepends@debug")

(assuming pak works for this), and then after restarting R:

pkgdepends::new_pkg_installation_proposal("cli", config = list(dependencies = FALSE))$resolve()

and then send me the output?

stephenashton-dhsc commented 1 month ago

> devtools::install_github("r-lib/pkgdepends", ref = "debug")
Downloading GitHub repo r-lib/pkgdepends@debug
── R CMD build ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/tmp/RtmpohPr1e/remotes56ec4696d43f3/r-lib-pkgdepends-12c9baa/DESCRIPTION’ (509ms)
─  preparing ‘pkgdepends’:
✔  checking DESCRIPTION meta-information ...
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  building ‘pkgdepends_0.7.2.9000.tar.gz’

Installing package into ‘/data/users/hemt/R-library’
(as ‘lib’ is unspecified)
* installing *source* package ‘pkgdepends’ ...
** using staged installation
*** Running ./configure
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (pkgdepends)
> pkgdepends::new_pkg_installation_proposal("cli", config = list(dependencies = FALSE))$resolve()
Error in rec$remote[[1]] : subscript out of boundsard (CRAN/BioC) packages
> traceback()
6: stop(priv$value)
5: asNamespace("pkgcache")$synchronise(...)
4: synchronise(self$async_resolve())
3: pkgplan_resolve(self, private)
2: private$plan$resolve()
1: pkgdepends::new_pkg_installation_proposal("cli", config = list(dependencies = FALSE))$resolve()

stephenashton-dhsc commented 1 month ago

I restarted R as it doesn't seem to have changed. Looks like it's giving more info now:

> pkgdepends::new_pkg_installation_proposal("cli", config = list(dependencies = FALSE))$resolve()
✔ Loading metadata database ... done
<async_rejected in (function (e) {    recerror <<- e    recerror$aframe <<- recerror$aframe %||% find_async_data_frame()    recerror$calls <<- recerror$calls %||% sys.calls()    if (is.null(recerror[["call"]]))         recerror[["call"]] <<- sys.call()    recerror$parents <<- recerror$parents %||% sys.parents()    recerror[names(info)] <<- info    handler <- getOption("async.error")    if (is.function(handler))         handler()})(structure(list(message = "process exited with non-zero status",     call = NULL, data = list(status = 1L, stdout = "rpm: /usr/local/packages/xz/5.2.3/lib/liblzma.so.5: version `XZ_5.1.2alpha' not found (required by /lib64/librpmio.so.3)\n",         stderr = "rpm: /usr/local/packages/xz/5.2.3/lib/liblzma.so.5: version `XZ_5.1.2alpha' not found (required by /lib64/librpmio.so.3)\n",         timeout = FALSE)), class = c("async_rejected", "simpleError", "error", "condition"))): process exited with non-zero status>
# A data frame: 1 × 6
  ref   remote         status direct async_id started_at
<chr> <list>         <chr>  <lgl>     <int> <dttm>
1 cli   <rmt_rf_s [6]> NA     TRUE          1 2024-05-03 12:03:44
# A data frame: 0 × 6
# ℹ 6 variables: ref <chr>, remote <list>, status <chr>, direct <lgl>,
#   async_id <int>, started_at <dttm>
Called from: private[[which]](value, private$resolve)

gaborcsardi commented 1 month ago

Oh, seems like there is an issue with system requirements installation. Try this:

Sys.setenv("PKG_SYSREQS" = "false")
pak::pkg_install("ggplot2")

stephenashton-dhsc commented 1 month ago

Unfortunately that doesn't seem to have changed anything (even after a restart of R):

> Sys.getenv("PKG_SYSREQS")
[1] "false"
> pak::pkg_install("ggplot2")
✔ Loading metadata database ... done
Error:  1  deps for  0/1  pkgs [⠹] Resolving standard (CRAN/BioC) packages
! error in pak subprocess
Caused by error in `rec$remote[[1]]`:
! subscript out of bounds
Type .Last.error to see the more details.
> pkgdepends::new_pkg_installation_proposal("cli", config = list(dependencies = FALSE))$resolve()
✔ Loading metadata database ... done
<async_rejected in (function (e) {    recerror <<- e    recerror$aframe <<- recerror$aframe %||% find_async_data_frame()    recerror$calls <<- recerror$calls %||% sys.calls()    if (is.null(recerror[["call"]]))         recerror[["call"]] <<- sys.call()    recerror$parents <<- recerror$parents %||% sys.parents()    recerror[names(info)] <<- info    handler <- getOption("async.error")    if (is.function(handler))         handler()})(structure(list(message = "process exited with non-zero status",     call = NULL, data = list(status = 1L, stdout = "rpm: /usr/local/packages/xz/5.2.3/lib/liblzma.so.5: version `XZ_5.1.2alpha' not found (required by /lib64/librpmio.so.3)\n",         stderr = "rpm: /usr/local/packages/xz/5.2.3/lib/liblzma.so.5: version `XZ_5.1.2alpha' not found (required by /lib64/librpmio.so.3)\n",         timeout = FALSE)), class = c("async_rejected", "simpleError", "error", "condition"))): process exited with non-zero status>
# A data frame: 1 × 6
  ref   remote         status direct async_id started_at
<chr> <list>         <chr>  <lgl>     <int> <dttm>
1 cli   <rmt_rf_s [6]> NA     TRUE          1 2024-05-03 12:17:51
# A data frame: 0 × 6
# ℹ 6 variables: ref <chr>, remote <list>, status <chr>, direct <lgl>,
#   async_id <int>, started_at <dttm>
Called from: private[[which]](value, private$resolve)

gaborcsardi commented 1 month ago

Hmmm, I think the root of the problem is that this is broken for you (from a shell):

rpm -qa

Hopefully this works:

Sys.setenv(PKG_SYSREQS_PLATFORM = "unknown")
pak::pkg_install("ggplot2")

(Closed accidentally, sorry.)

stephenashton-dhsc commented 1 month ago

That got it! Massive thanks :)

I'll flag the issue with rpm -qa with our system admin (as you're correct and it's not working)

r-lib / pak

Error in `rec$remote[[1]]`: subscript out of bounds #622