push-button deploy of renv project fails

[aronatkins]

── Preparing for deployment ────────────────────────────────────────────────────
✔ Deploying "shiny-renv" to "server: XXXXX / username: XXXXX"
ℹ Creating application on server...
✔ Created application with id XXXXX
ℹ Bundling 3 files: .Rprofile, app.R, and renv.lock
ℹ Capturing R dependencies from renv.lock
- Operation canceled.
Execution halted

When run in an interactive context, we get prompted for input. This example shows manifest generation, but similar output occurs with rsconnect::deployApp():

> rsconnect::writeManifest()
ℹ Capturing R dependencies from renv.lock
What do you want to do? 

1: Snapshot, just using the currently installed packages.
2: Install the packages, then snapshot.
3: Cancel, and resolve the situation on your own.

Selection: 

The project in-use was a very simple Shiny application that used a newly created renv configuration to track its dependencies.

[aronatkins]

Create the following Shiny application in a new directory:

# app.R
library(shiny)

ui <- fluidPage(h1("Hello from Shiny"),
                h2("Using renv in development."))
server <- function(input, output) {}

shinyApp(ui, server)

Open the RStudio IDE using that directory as a project.

From the R console, run:

renv::init()
# R session is restarted
renv::restore()
#> - The library is already synchronized with the lockfile.

Create a manifest from this project state:

> rsconnect::writeManifest()
ℹ Capturing R dependencies from renv.lock
What do you want to do? 

1: Snapshot, just using the currently installed packages.
2: Install the packages, then snapshot.
3: Cancel, and resolve the situation on your own.

Selection: 

This prompt does not present very much information and in a non-interactive context (push-button deploy), causes an error.

[aronatkins]

Workaround: Exclude the renv.lock from consideration. For example, with the simple Shiny application, we can include only the app.R:

rsconnect::writeManifest(appFiles="app.R")
#> ℹ Capturing R dependencies with renv

This strategy also works with push-button deployment: Exclude the renv.lock from the deployment.

Note: folks will likely also need the fix to https://github.com/rstudio/rsconnect/issues/916, which can be installed from GitHub:

remotes::install_github("rstudio/rsconnect")

Alternatively, folks can install the previous version of rsconnect:

remotes::install_version("rsconnect", "0.8.29")

[aronatkins]

This problem may have been observed externally: https://stackoverflow.com/questions/76713367/unable-to-deploy-to-shinyapps-io/76716053#76716053

[aronatkins]

This may be (partially) related to https://github.com/rstudio/renv/pull/1545, which is resolved using the development version of renv:

remotes::install_github("rstudio/renv")

[aronatkins]

With the latest version of renv, interactive deploys appear functional, but non-interactive (push-button) deploys show errors like:

ℹ Capturing R dependencies from renv.lock
Error in read.dcf(path) : cannot open the connection
Calls: <Anonymous> ... lapply -> FUN -> as.list -> as.data.frame -> read.dcf
In addition: Warning message:
In read.dcf(path) :
  cannot open compressed file '', probable reason 'No such file or directory'
Execution halted

[aronatkins]

In an interactive context, we see the temporarily restored renv_library contain all of the target packages:

askpass, base64enc, bslib, cachem, cli, commonmark, crayon, curl, digest, ellipsis, fastmap, fontawesome, fs, glue, htmltools, httpuv, jquerylib, jsonlite, later, lifecycle, magrittr, memoise, mime, openssl, packrat, promises, R6, rappdirs, Rcpp, renv, rlang, rsconnect, rstudioapi, sass, shiny, sourcetools, sys, withr, xtable, yaml

Within the deploy pane, we see only:

renv, rsconnect

As a result, package_record is unable to discover a DESCRIPTION file for many of the referenced packages, causing the read.dcf error:

https://github.com/rstudio/rsconnect/blob/647bb4def63db199be547c0c48603bc511db3686/R/bundlePackage.R#L100-L103

These package lookups happen immediately after the renv::restore:

https://github.com/rstudio/rsconnect/blob/647bb4def63db199be547c0c48603bc511db3686/R/bundlePackageRenv.R#L44-L49

[aronatkins]

When deployment happens within the deploy pane:

.libPaths():
    /Users/aron/dev/rstudio/standalone/shiny-renv/renv/library/R-4.3/x86_64-apple-darwin20,
    /Users/aron/Library/Caches/org.R-project.R/R/renv/sandbox/R-4.3/x86_64-apple-darwin20/84ba8b13,
    /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library 

When deployment happens from the R console:

.libPaths():
    /Users/aron/dev/rstudio/standalone/shiny-renv/renv/library/R-4.3/x86_64-apple-darwin20,
    /Users/aron/Library/Caches/org.R-project.R/R/renv/sandbox/R-4.3/x86_64-apple-darwin20/84ba8b13 

The .libPaths() differences in the two contexts are probably causing a difference in renv::restore behavior.

[kevinushey]

For reference, I think the issue here is:

1. When an renv project is loaded, renv will activate the R sandbox, effectively replacing .Library with the sandbox library path,
2. When RStudio launches an R sub-process, it tries to forward the R library paths via the R_LIBS environment variable,
3. However, because the renv project is not loaded (presumedly?) during deployment, R ends up restoring the default version of .Library onto the library paths.

Maybe we need to make sure RStudio is sourcing the application's .Rprofile on startup here?

https://github.com/rstudio/rstudio/blob/9d3b5a4f4c5abbceee1e91efb7bb90ec4727018d/src/cpp/session/modules/SessionRSConnect.cpp#L294

That is, maybe we don't want to be vanilla if the application has a .Rprofile?

[kevinushey]

Either that, or rsconnect needs to repair the library paths post-hoc when deployApp() is called, which feels kind of gross and not really rsconnect's responsibility.

[hadley]

With latest dev renv, I created app.R then did:

renv::init(repos = "https://cloud.r-project.org")
renv::install("rsconnect")
rsconnect::writeManifest()

That worked, so then I push-button deployed to colorado:

── Preparing for deployment ────────────────────────────────────────────────────
✔ Deploying "testrsc" to "server: colorado.posit.co / username: hadley"
ℹ Creating application on server...
✔ Created application with id 17348
ℹ Bundling 3 files: .Rprofile, app.R, and renv.lock
ℹ Capturing R dependencies from renv.lock
✔ Found 31 dependencies
✔ Created 17,928b bundle
ℹ Uploading bundle...
✔ Uploaded bundle with id 77339

And that works for me 😬

[aronatkins]

@hadley - what version of RStudio? My testing has been with the latest dailies; I'm using 2023.09.0-daily+235 right now and seeing packages missing from the renv_library.

If you install my aron-debugging-records rsconnect branch, the deploy will display what is in the library and the .libPaths().

[hadley]

Here's even more of a reprex, in case it's something that we're doing differently:

pak::pak("rstudio/renv")
usethis::create_project("~/Desktop/test-push-button-deploy")

# in new RStudio
writeLines(con = "app.R", c(
  "library(shiny)",
  "ui <- fluidPage('hello')",
  "server <- function(input, output) {}",
  "shinyApp(ui, server)"
))
renv::init(repos = "https://cloud.r-project.org")
# R restarts
renv::install("rstudio/rsconnect@aron-debugging-records")
rsconnect::writeManifest()
unlink("manifest.json")

# open app.R
# click publish button

And here's the output:

── Preparing for deployment ────────────────────────────────────────────────────
✔ Re-deploying "test-push-button-deploy" to "server: colorado.posit.co / username: hadley"
ℹ Looking up application with id "17350"...
✔ Found application <https://colorado.posit.co/rsc/content/11c007c3-10aa-4f7f-8f16-cd9dc8283fea/>
ℹ Bundling 3 files: .Rprofile, app.R, and renv.lock
ℹ Capturing R dependencies from renv.lock
PACKAGES: R6, Rcpp, askpass, base64enc, bslib, cachem, cli, commonmark, crayon, curl, digest, ellipsis, fastmap, fontawesome, fs, glue, htmltools, httpuv, jquerylib, jsonlite, later, lifecycle, magrittr, memoise, mime, openssl, packrat, promises, rappdirs, renv, rlang, rsconnect, rstudioapi, sass, shiny, sourcetools, sys, withr, xtable, yaml 
LIBRARY: /tmp/Rtmp71Drgm/file14b84198970a6/renv_library 
LIBRARY FILES: askpass, base64enc, bslib, cachem, cli, commonmark, crayon, curl, digest, ellipsis, fastmap, fontawesome, fs, glue, htmltools, httpuv, jquerylib, jsonlite, later, lifecycle, magrittr, memoise, mime, openssl, packrat, promises, R6, rappdirs, Rcpp, renv, rlang, rsconnect, rstudioapi, sass, shiny, sourcetools, sys, withr, xtable, yaml 
.libPaths(): /Users/hadleywickham/Desktop/test-push-button-deploy/renv/library/R-4.2/aarch64-apple-darwin20, /Users/hadleywickham/Library/Caches/org.R-project.R/R/renv/sandbox/R-4.2/aarch64-apple-darwin20/fd29d0b8, /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/library 
✔ Found 40 dependencies
✔ Created 21,812b bundle
ℹ Uploading bundle...
✔ Uploaded bundle with id 77345

I'm using RStudio 2023.06.0+421

[aronatkins]

Those exact steps show the following deploy pane failure:

── Preparing for deployment ────────────────────────────────────────────────────
✔ Deploying "test-push-button-deploy" to "server: rsc.radixu.com / username: aron"
ℹ Creating application on server...
✔ Created application with id 20844
ℹ Bundling 3 files: .Rprofile, app.R, and renv.lock
ℹ Capturing R dependencies from renv.lock
PACKAGES: R6, Rcpp, base64enc, bslib, cachem, cli, commonmark, crayon, digest, ellipsis, fastmap, fontawesome, fs, glue, htmltools, httpuv, jquerylib, jsonlite, later, lifecycle, magrittr, memoise, mime, promises, rappdirs, renv, rlang, sass, shiny, sourcetools, withr, xtable 
LIBRARY: /var/folders/8j/pt4fjhkj11d3xd9344pdspmh0000gn/T//RtmpAU51pI/file3b3a35358a42/renv_library 
LIBRARY FILES:  
.libPaths(): /Users/aron/Desktop/test-push-button-deploy/renv/library/R-4.3/x86_64-apple-darwin20, /Users/aron/Library/Caches/org.R-project.R/R/renv/sandbox/R-4.3/x86_64-apple-darwin20/84ba8b13, /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library 
Error in read.dcf(path) : cannot open the connection
Calls: <Anonymous> ... lapply -> FUN -> as.list -> as.data.frame -> read.dcf
In addition: Warning message:
In read.dcf(path) :
  cannot open compressed file '', probable reason 'No such file or directory'
Execution halted

[rsh52]

Also running into this for an renv-equipped project I'm deploying to Posit Connect. Rolling the package version back to 0.8.29 went through successfully.

[aronatkins]

@rsh52 - Are you able to test using the development versions of rsconnect and renv? Does the result change?

renv::install("rstudio/rsconnect")
renv::install("rstudio/renv")

[aronatkins]

I see the same read.dcf error with multiple RStudio versions, including 2023.06.0+421. 😠

[kevinushey]

I think I understand why we were running into issues. There's a couple of things coming together to cause this:

1. @aronatkins only has a single library path; that is, all R packages get installed into the default library .Library, not a user library;
2. A recent change to renv causes us to consider .Library during restore: https://github.com/rstudio/renv/blob/75c75dc51071a1a916ace5ec6f00ccfac6cd4b51/R/libpaths.R#L189-L201
3. RStudio launches the R process that handles the deployment using --vanilla: https://github.com/rstudio/rstudio/blob/c946f30a7ca642358d32fc487f4d4c2f8bfef6d2/src/cpp/session/modules/SessionRSConnect.cpp#L294
4. Because we run with --vanilla, the renv sandbox is not re-activated, and so the default library paths remain on the .libPaths;
5. Because restore() now also checks whether the required packages are available in .Library, and sees them there, it decides to do nothing (doesn't install packages into the requested library path).

Changing this in RStudio will probably be hard, so the most straightforward fix might be for renv to try to recover in this scenario. That is, if renv sees both the sandbox + the default library on the library paths, then activate the sandbox so that the system library is again hidden.

Update(@aronatkins): updated the renv link

[rsh52]

@rsh52 - Are you able to test using the development versions of rsconnect and renv? Does the result change?

renv::install("rstudio/rsconnect")
renv::install("rstudio/renv")

I started running into an issue where renv wouldn't let me install packages normally, I instead had to install via utils::install.packages() as described here (which funny enough @kevinushey answered). I updated back to rsconnect 1.0, and instead of using button deployment I ran rsconnect::writeManifest() + rsconnect::deployApp() which seems to have worked. The button deployment still failed though.

[aronatkins]

While debugging with @kevinushey, we were able to successfully deploy by copying .Rprofile to .rsconnect_profile, which activated renv from within the RStudio deploy pane "vanilla" session. This helped us understand the library path problem.

https://github.com/rstudio/rsconnect/blob/af467ef9f9a47c274e9ab391f9afb9d0a3047c6c/R/deployApp.R#L617-L624

[aronatkins]

https://github.com/rstudio/renv/issues/1565 looks to track / resolve the problem I was seeing locally.

[aronatkins]

With https://github.com/rstudio/renv/issues/1565 resolved, we believe that this issue can be closed. Folks will need to use the development versions of rsconnect and renv until the next CRAN release of each.

remotes::install_github("rstudio/rsconnect")
remotes::install_github("rstudio/renv")

[kmasiello]

Hi @aronatkins, The basic Shiny app is still failing to deploy when renv.lock is included in the list of files. Tested with 1.0.1 and 1.0.1.9000.

[aronatkins]

Tracking possible lingering (similar) problems with https://github.com/rstudio/rsconnect/issues/925.

[kmasiello]

successfully deployed app with renv.lock with renv v 1.0.0.9000 and rsconnect v 1.0.1.