search

rstudio / rsconnect

Publish Shiny Applications, RMarkdown Documents, Jupyter Notebooks, Plumber APIs, and more
http://rstudio.github.io/rsconnect/
134 stars 83 forks source link

Issue deploying bioconductor packages #1045

Open le-raman opened 8 months ago

le-raman commented 8 months ago

I've set:

options(repos = c(BiocManager::repositories()))

So:

> options("repos")
$repos
                                                BioCsoft                                                  BioCann 
           "https://bioconductor.org/packages/3.18/bioc" "https://bioconductor.org/packages/3.18/data/annotation" 
                                                 BioCexp                                            BioCworkflows 
"https://bioconductor.org/packages/3.18/data/experiment"       "https://bioconductor.org/packages/3.18/workflows" 
                                               BioCbooks 
          "https://bioconductor.org/packages/3.18/books"

But deployment fails on certain bioconductor packages (not all), e.g., BiocVersion:

── Preparing for deployment ────────────────────────────────────────────────────
ℹ Capturing R dependencies from renv.lock
✔ Found 190 dependencies
✔ Created 730,925b bundle
ℹ Uploading bundle...
✔ Uploaded bundle with id 473
── Deploying to server ─────────────────────────────────────────────────────────
Building Shiny application...
Bundle created with R version 4.3.2 is compatible with environment Local with R version 4.1.2 from /opt/R/4.1.2/bin/R 
Bundle requested R version 4.3.2; using /opt/R/4.1.2/bin/R which has version 4.1.2
Performing manifest.json to packrat transformation.
[rsc-session] Content GUID: 283fb64a-0881-4d45-bdd8-c692ee3ec47d
[rsc-session] Content ID: 144
[rsc-session] Bundle ID: 473
[rsc-session] Job Key: tjz1SC7CQm0Nbm7i
Running on host: ip-10-128-2-199
Linux distribution: Ubuntu 20.04.6 LTS (focal)
Running as user: uid=997(rstudio-connect) gid=998(rstudio-connect) groups=998(rstudio-connect)
Connect version: 2024.01.0
LANG: C.UTF-8
Working directory: /opt/rstudio-connect/mnt/app
Using R 4.1.2
R.home(): /opt/R/4.1.2/lib/R
Using user agent string: 'RStudio R (4.1.2 x86_64-pc-linux-gnu x86_64 linux-gnu)' 
# Validating R library read / write permissions --------------------------------
Using R library for packrat bootstrap: /opt/rstudio-connect/mnt/R/4.1.2
# Validating managed packrat installation --------------------------------------
Vendored packrat archive: /opt/rstudio-connect/ext/R/packrat_0.9.2.9000_70625806c44bda42a7f3aeaa92ee65542cc590be.tar.gz
Vendored packrat SHA: 70625806c44bda42a7f3aeaa92ee65542cc590be
Managed packrat SHA:  70625806c44bda42a7f3aeaa92ee65542cc590be
Managed packrat version: 0.9.2.9000
Managed packrat is up-to-date.
# Validating packrat cache read / write permissions ----------------------------
Using packrat cache directory: /opt/rstudio-connect/mnt/packrat/4.1.2
# Setting packrat options and preparing lockfile -------------------------------
Audited package hashes with local packrat installation.
# Resolving R package repositories ---------------------------------------------
Received repositories from published content:
- BioCann = "https://bioconductor.org/packages/3.18/data/annotation"
- BioCexp = "https://bioconductor.org/packages/3.18/data/experiment"
- BioCsoft = "https://bioconductor.org/packages/3.18/bioc"
- BioCworkflows = "https://bioconductor.org/packages/3.18/workflows"
- CRAN = "https://p3m.dev/cran/latest"
# Installing required R packages with `packrat::restore()` ---------------------
Warning in packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 4.3.2
Installing BH (1.84.0-0) ... 
    OK (symlinked cache)
Installing BiocGenerics (0.48.1) ... 
    OK (symlinked cache)
Installing BiocManager (1.30.22) ... 
    OK (symlinked cache)
Installing BiocVersion (3.18.1) ... 
curl: HTTP 200 https://bioconductor.org/packages/3.18/data/annotation/src/contrib/PACKAGES.rds
curl: HTTP 200 https://bioconductor.org/packages/3.18/data/experiment/src/contrib/PACKAGES.rds
curl: HTTP 200 https://bioconductor.org/packages/3.18/bioc/src/contrib/PACKAGES.rds
curl: HTTP 200 https://bioconductor.org/packages/3.18/workflows/src/contrib/PACKAGES.rds
curl: HTTP 200 https://p3m.dev/cran/latest/src/contrib/PACKAGES.rds
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/annotation/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/annotation/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/annotation/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/annotation/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/annotation/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/experiment/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/experiment/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/experiment/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/experiment/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/data/experiment/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/bioc/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/bioc/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/bioc/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/bioc/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/bioc/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/workflows/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/workflows/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/workflows/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/workflows/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://bioconductor.org/packages/3.18/workflows/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/BiocVersion/BiocVersion_3.18.1.tar.gz
FAILED
Error in getSourceForPkgRecord(pkgRecord, srcDir(project), availablePackagesSource(repos = repos), : Failed to retrieve package sources for BiocVersion 3.18.1 from CRAN (internet connectivity issue?)

Unable to fully restore the R packages associated with this deployment.
Please review the preceding messages to determine which package
encountered installation difficulty and the cause of the failure.

An error occurred while building your content.
This link offers advice for this kind of error:
    https://docs.posit.co/connect/user/troubleshooting/#r-package-not-available
Build error: An error occurred while building your content. (Error code: r-package-not-available)
── Deployment complete ─────────────────────────────────────────────────────────
✖ Deployment failed with error: An error occurred while building your content. (Error code: r-package-not-available)

Any ideas what might cause this? I've included my env lock file if this would be relevant. I'm on rsconnect version 1.2.1.

Thanks!

env.lock.txt

le-raman commented 8 months ago

Another update:

The same error occurs for, e.g., S4Arrays. S4Arrays is indeed not part of the archive where it's looking for the installation files (https://bioconductor.org/packages/3.18/bioc/src/contrib/Archive/).

In case of BiocVersion, it is present, https://bioconductor.org/packages/3.18/bioc/src/contrib/Archive/BiocVersion/, but not version 3.18.1.

I've tried changing options(repos = ...) to several other repositories I could find on Bioconductor, but no luck.

le-raman commented 8 months ago

Also tried downgrading to Bioconductor 3.17, but no luck either:

Installing S4Arrays (1.2.0) ... 
curl: HTTP 200 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/annotation/src/contrib/PACKAGES.rds
curl: HTTP 200 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/experiment/src/contrib/PACKAGES.rds
curl: HTTP 200 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/bioc/src/contrib/PACKAGES
curl: HTTP 200 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/bioc/src/contrib/PACKAGES.gz
curl: HTTP 200 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/workflows/src/contrib/PACKAGES.rds
curl: HTTP 200 https://p3m.dev/cran/latest/src/contrib/PACKAGES.rds
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/annotation/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/annotation/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/annotation/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/annotation/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/annotation/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/experiment/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/experiment/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/experiment/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/experiment/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/data/experiment/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/bioc/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/bioc/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/bioc/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/bioc/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/bioc/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/workflows/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/workflows/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/workflows/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/workflows/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
curl: HTTP 404 https://mghp.osn.xsede.org/bir190004-bucket01/archive.bioconductor.org/packages/3.17/workflows/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
curl: (22) The requested URL returned error: 404 
curl: HTTP 404 https://p3m.dev/cran/latest/src/contrib/Archive/S4Arrays/S4Arrays_1.2.0.tar.gz
FAILED
Error in getSourceForPkgRecord(pkgRecord, srcDir(project), availablePackagesSource(repos = repos), : Failed to retrieve package sources for S4Arrays 1.2.0 from CRAN (internet connectivity issue?)

Unable to fully restore the R packages associated with this deployment.
Please review the preceding messages to determine which package
encountered installation difficulty and the cause of the failure.

An error occurred while building your content.
This link offers advice for this kind of error:
    https://docs.posit.co/connect/user/troubleshooting/#r-package-not-available
Build error: An error occurred while building your content. (Error code: r-package-not-available)
── Deployment complete ─────────────────────────────────────────────────────────
✖ Deployment failed with error: An error occurred while building your content. (Error code: r-package-not-available)

Any help greatly appreciated!

le-raman commented 8 months ago

And here is a minimal reproducible example:

Clean R installation:

if (!require("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install(version = "3.18")

App:

library(shiny)
library(BiocVersion)

ui <- fluidPage(
  textOutput("output")
)

server <- function(input, output, session) {
  output$output <- renderText({ 
    paste0("test")
  })
}

shinyApp(ui, server)
kevinushey commented 8 months ago

I think I know what's going on. You're trying to use Bioconductor 3.18, but R 4.1.2 is being used during deployment on the Connect server. That version of Bioconductor and its packages requires R (>= 4.3.0), so the packages you're requesting aren't available for the version of R in use.

le-raman commented 8 months ago

Thanks for the response!

The rsconnect server indeed runs R 4.1.2. The message at the start of deployment "Bundle created with R version 4.3.2 is compatible with environment Local with R version 4.1.2" made me believe this was not an issue. I'll try different R versions and let you know how that goes.

le-raman commented 8 months ago

I've installed R 4.3.2 in parallel to R 4.1.2 on our Posit Connect server and rebooted; app is being deployed as we speak, but I can already confirm it retrieved the BiocVersion installation files successfully.

I've configured server.RVersionMatching = exact to make sure I don't face similar issues in the future.

Thanks a lot for your help!! I'll leave the issue open for now, in case you want to tackle the deployment messages prompted to the user.

kevinushey commented 8 months ago

cc @aronatkins: is there a place in rsconnect or Connect where the version of Bioconductor being used could be validated against the version of R requested? (For example, if a manifest requests Bioconductor 3.18, then require R 4.3.0 or newer.)

The BiocManager package has this, which can help map R versions to Bioconductor versions:

> BiocManager:::.version_map()
   Bioc    R  BiocStatus
1   1.6  2.1 out-of-date
2   1.7  2.2 out-of-date
3   1.8  2.3 out-of-date
4   1.9  2.4 out-of-date
5   2.0  2.5 out-of-date
6   2.1  2.6 out-of-date
7   2.2  2.7 out-of-date
8   2.3  2.8 out-of-date
9   2.4  2.9 out-of-date
10  2.5 2.10 out-of-date
11  2.6 2.11 out-of-date
12  2.7 2.12 out-of-date
13  2.8 2.13 out-of-date
14  2.9 2.14 out-of-date
15 2.10 2.15 out-of-date
16 2.11 2.15 out-of-date
17 2.12  3.0 out-of-date
18 2.13  3.0 out-of-date
19 2.14  3.1 out-of-date
20  3.0  3.1 out-of-date
21  3.1  3.2 out-of-date
22  3.2  3.2 out-of-date
23  3.3  3.3 out-of-date
24  3.4  3.3 out-of-date
25  3.5  3.4 out-of-date
26  3.6  3.4 out-of-date
27  3.7  3.5 out-of-date
28  3.8  3.5 out-of-date
29  3.9  3.6 out-of-date
30 3.10  3.6 out-of-date
31 3.11  4.0 out-of-date
32 3.12  4.0 out-of-date
33 3.13  4.1 out-of-date
34 3.14  4.1 out-of-date
35 3.15  4.2 out-of-date
36 3.16  4.2 out-of-date
37 3.17  4.3 out-of-date
38 3.18  4.3     release
39 3.19  4.4       devel
40 3.19  4.5      future

Or alternatively, this information can also be queried from https://bioconductor.org/config.yaml.