rsconnect fails to deploy app when necessary package is installed from github w/ renv in root

[richarddmorey]

I've been trying to debug a problem deploying a shiny app to shinyapps.io that involves a package installed from github. I believe I've narrowed the issue down to something having to do with renv (see here: https://github.com/r-lib/pillar/issues/252#issuecomment-877728907) and @klrmlr has requested that I raise it here.

Description

Here's the gist: I have two versions of a package on github. They are identical except one has renv in the package root, and in the other I have removed renv. The package is an extremely simple MWE: it has one exported function and one dependency (dplyr). See here: https://github.com/richarddmorey/shinyapps_problem (the master branch has renv in the root).

When I deploy a simple Rmd file to shinyapps.io that depends on the version of the package with renv in the project root, it fails (log). When I deploy the same Rmd file that depends on a version of the package that does NOT have renv defined in the project root (branch) the deploy succeeds (log).

Session info is here: session info

Steps to reproduce:

1. Download the renv_test branch of the repository (https://github.com/richarddmorey/shinyapps_problem/tree/renv_test)
2. Open the project, and then demo_code/R/demo_code.R
3. Change the rsconnect username to whatever your username is
4. Run the code in demo_code/R/demo_code.R

What should happen:

The first deployment (with richarddmorey/shinyapps_problem@master installed) should lead to this failure; the second deployment (with richarddmorey/shinyapps_problem@test3 installed) should lead to success.

You can verify that richarddmorey/shinyapps_problem@test3 only differs from richarddmorey/shinyapps_problem@master with respect to renv by looking at the two additional commits.

To confirm that this is not a problem with the order in which I am installing the two identical packages, you can run the code in demo_code/R/demo_code_reverse_order.R (which fails in the same way).

I do not understand why deploying an Rmd based on the version of the package that has renv in the root fails; as far as I can tell, once they're installed they look similar on disk. I thought it might have to do with the renv .Rprofile (see this comment by @krlmlr) but I'm kind of baffled to be honest.

[kevinushey]

You probably need to make sure the renv autoloader (in the project .Rprofile) is not included in the upload. My suspicion is that renv is trying to bootstrap itself and is mutating the library paths in a way that shinyapps.io does not expect.

[richarddmorey]

You probably need to make sure the renv autoloader (in the project .Rprofile) is not included in the upload. My suspicion is that renv is trying to bootstrap itself and is mutating the library paths in a way that shinyapps.io does not expect.

When you say the upload, do you mean the installation of the dependency from github or the deployment using rsconnect::deployApp? I'm only deploying the Rmd file (using the argument appFiles), not the .Rprofile:

https://github.com/richarddmorey/shinyapps_problem/blob/ce129e998fd87b2bae771eae7e68b9798903606f/demo_code/R/demo_code.R#L20

The problem seems to be the contents of the dependency repository. Would an .Rprofile in the package root for a github dependency get installed somehow when the Rmd file is deployed? The documentation for writing R extensions doesn't seem to mention .Rprofile as something that is used in the package installation.

[kevinushey]

The problem seems to be the contents of the dependency repository. Would an .Rprofile in the package root for a github dependency get installed somehow when the Rmd file is deployed? The documentation for writing R extensions doesn't seem to mention .Rprofile as something that is used in the package installation.

On startup, R will (unless launched with --vanilla or, I think, --no-init-file) attempt to read a .Rprofile either in the current working directory, or in the user's home directory. My guess is that during the R CMD INSTALL attempt, because R is launched within the directory containing that .Rprofile, it's being read and initialized on start.

[2021-07-11T21:43:52.261648488+0000] Building R package: flexTeaching (0.1)
/mnt/packages/build /mnt
# Bootstrapping renv 0.13.2 --------------------------------------------------
* Downloading renv 0.13.2 ... OK (downloaded source)
* Installing renv 0.13.2 ... Done!
* Successfully installed and loaded renv 0.13.2.
Error: .onLoad failed in loadNamespace() for 'pillar', details:
  call: loadNamespace(x)
  error: there is no package called ‘crayon’
Execution halted
Warning in untar2(tarfile, files, list, exdir, restore_times) :
  skipping pax global extended headers
* installing to library ‘/opt/R/4.1.0/lib/R/library’
* installing *source* package ‘flexTeaching’ ...
** using non-staged installation via StagedInstall field
** R
** inst
** byte-compile and prepare package for lazy loading
ERROR: lazy loading failed for package ‘flexTeaching’
* removing ‘/opt/R/4.1.0/lib/R/library/flexTeaching’
GET /v1/tasks/?filter=account_id:26575&filter=parent_id:969702251&count=100&offset=0 419ms
################################# End Task Log ################################# 

So it seems like it's an artefact of how packages are built and installed on shinyapps.io. Could you raise this on https://community.rstudio.com/c/shiny/8, so that the shinyapps.io folks can try and take a look?

[richarddmorey]

Could you raise this on https://community.rstudio.com/c/shiny/8, so that the shinyapps.io folks can try and take a look?

I raised it months ago with no success (both on the community and with Rstudio support), but now that I have a workaround and so can manipulate the problem it may be clearer to them what is happening. I re-opened my ticket earlier today, so hopefully I hear back.

[kevinushey]

Can you also share a link to the associated issue here?

[richarddmorey]

Can you also share a link to the associated issue here?

Just checked and I misremembered; it was someone else that posted the issue (https://community.rstudio.com/t/failed-to-deployapp-error-there-is-no-package-called-crayon/89921); I only "liked" it.

Although the error they were getting is the same as mine, rereading my private correspondence with the author it appears now that it probably wasn't the same issue given that he fixed it by adding pillar and crayon to "Imports" (https://github.com/CRI-iAtlas/iatlas.modules/commit/95a255e26509b452f1ffe7c5f4310f4f19ded712). I had to work around it in a different way.

I'm waiting to hear back again from RStudio support; depending on what I hear from them, I'll post something on the community forums.

[kippandrew]

@richarddmorey I suspect there is a .RProfile in your bundle that is being uploaded to shinyapps.io which is triggering this behavior. I can confirm this if you send me an APP ID or BUNDLE ID that failed to build. That said I think @kevinushey's point is right, there is likely an issue on the shinyapps.io backend where we should be ignoring these files with --vanilla. I'm actually kind of surprised we don't do this already.

[richarddmorey]

@richarddmorey I suspect there is a .RProfile in your bundle that is being uploaded to shinyapps.io which is triggering this behavior. I can confirm this if you send me an APP ID or BUNDLE ID that failed to build.

Sure; I just attempted (and failed) to deploy app ID 4386754 (app name: solve_test, username richarddmorey).

Where would the .Rprofile come from in this case? In both the failure and success, I'm passing the same arguments to rsconnect::deployApp() which only have reference a singe Rmd file (using the appFiles argument). So if there is an .Rprofile file, I'm not sure how it would have gotten there.

[kevinushey]

My understanding is that the .Rprofile is part of the GitHub repository for the package being installed; not the bundle itself. Is that correct?

While R CMD INSTALL is running, some child R processes will be launched using the package root as the working directory. For example:

kevinushey@cdrv-2:~/scratch
$ cat testing/.Rprofile
print("Hello from .Rprofile")
kevinushey@cdrv-2:~/scratch
$ R CMD INSTALL testing
* installing to library ‘/Users/kevinushey/Library/R/x86_64/4.1/library’
* installing *source* package ‘testing’ ...
** using staged installation
** inst
** help
No man pages found in package  ‘testing’
*** installing help indices
** building package indices
[1] "Hello from .Rprofile"
** testing if installed package can be loaded from temporary location
[1] "Hello from .Rprofile"
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
[1] "Hello from .Rprofile"
** testing if installed package keeps a record of temporary installation path
* DONE (testing)

That effect is alleviated if we run install as R --vanilla CMD INSTALL <...>.

[richarddmorey]

My understanding is that the .Rprofile is part of the GitHub repository for the package being installed; not the bundle itself. Is that correct?

Ah, yes, I just tried it and I see that this occurs even when you start R from a different folder (eg R CMD install ~/pkg/foo).

 ~ % R CMD install Documents/Projects/shinyapps_problem
* installing to library ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library’
* installing *source* package ‘flexTeaching’ ...
** using non-staged installation via StagedInstall field
** R
** inst
** byte-compile and prepare package for lazy loading
# Bootstrapping renv 0.13.2 --------------------------------------------------
* Downloading renv 0.13.2 ... OK (downloaded binary)
* Installing renv 0.13.2 ... Done!
* Successfully installed and loaded renv 0.13.2.
Error: .onLoad failed in loadNamespace() for 'pillar', details:
  call: loadNamespace(x)
  error: there is no package called ‘crayon’
Execution halted
ERROR: lazy loading failed for package ‘flexTeaching’
* removing ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library/flexTeaching’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library/flexTeaching’

...which is the error I am seeing in the shinyapps.io log.

Indeed, with --vanilla, it works fine:

 ~ % R --vanilla CMD install Documents/Projects/shinyapps_problem
* installing to library ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library’
* installing *source* package ‘flexTeaching’ ...
** using non-staged installation via StagedInstall field
** R
** inst
** byte-compile and prepare package for lazy loading
** help
No man pages found in package  ‘flexTeaching’
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (flexTeaching)

I have been installing using devtools::install_github but that doesn't have an issue. I guess it installs packages in a different way that isn't affected by the .Rprofile file being there? That was part of my great confusion: I installed with install_github specifically in order to use rsconnect, but then it failed on the server (I kind of assumed that shinyapps.io used install_github as well).

[richarddmorey]

For renv, is there a way to detect whether R was invoked with INSTALL or CHECK and not load in that case? (I tried commandArgs but that didn't seem to show INSTALL).

[kevinushey]

Good question. R sets some environment variables during install, e.g.

R_INSTALL_PKG           testing

And that gets set here by R during install:

https://github.com/wch/r-source/blob/0b3a0d303c6ddf2079352b07475bd67d9c2fde75/src/library/tools/R/install.R#L534-L535

so perhaps renv could / should check if that is set, and abort environment activation in that case.

[richarddmorey]

That effect is alleviated if we run install as R --vanilla CMD INSTALL <...>.

@kevinushey Will this be implemented on shinyapps.io (or maybe as an option in {rsconnect})?

[kippandrew]

@richarddmorey We have placed an issue on our backlog, but it hasn't been prioritized yet. I will make a note to update this ticket when that happens.

[kevinushey]

I think this issue should be resolved now -- there are mitigations both in renv and in shinyapps.io (although I'm not sure if the shinyapps changes have been deployed yet).