Packrat unable to restore package from private Gitlab

[DivadNojnarg]

Hi!

We (at Novartis) are currently trying to enable RStudio Connect users to deploy apps depending on private Gitlab packages. Currently, we face an issue.

Let me explain the process, we used ...

To reproduce the problem, I created a package, gitlabrsconnect located in an internal Novartis Gitlab. This package does nothing special and is installed with remotes (tried with both CRAN and devel versions):

remotes::install_gitlab(host= <NOVARTIS-GITLAB>, "repoOwner/gitlabrsconnect")

Then, within a RStudio server session, I run a sessioninfo to check whether everything is properly installed, for instance:

assertthat        0.2.1       2019-03-21 [5] CRAN (R 3.6.1)                           
 cli               1.1.0       2019-03-19 [5] CRAN (R 3.6.1)                           
 crayon            1.3.4       2017-09-16 [5] CRAN (R 3.6.1)                           
 gitlabrsconnect * 0.0.0.9000  2021-03-23 [1] gitlab (user/gitlabrsconnect@82570bb)
 packrat         * 0.5.0-32    2021-03-22 [1] Github (rstudio/packrat@6dd9d59)         
 remotes         * 2.2.0.9000  2021-03-23 [1] Github (r-lib/remotes@69a8bd1)           
 rstudioapi        0.13.0-9000 2021-02-25 [1] Github (rstudio/rstudioapi@8ed4d50)      
 sessioninfo       1.1.1       2018-11-05 [5] CRAN (R 3.6.1)                           
 withr             2.1.2       2018-03-15 [5] CRAN (R 3.6.1)                           
 yaml              2.2.0       2018-07-25 [5] CRAN (R 3.6.1)  

All custom packages are installed in a custom libPath ([1]), while other packages are taken from our internal CRAN snapshot ([5]). The output looks ok...

Then, I create a dummy project with a shiny app depending on the gitlabrsconnect dependency:

library(shiny)
library(gitlabrsconnect)
ui <- fluidPage()
server <- function(input, output, session) {}
shinyApp(ui, server)

If I use rsconnect::writeManifest(), I'll send up with an error regarding the Gitlab package source:

Warning: Unable to determine the repository for package gitlabrsconnect
Error:
* Unable to determine the location for some packages. Packages must come from a package
   repository like CRAN or a source control system. Check that options('repos') refers to a
   package repository containing the needed package versions.

Same thing, if I deploy the app with the publish button, the RStudio Connect server packrat (version 0.5.0.26, so not the very latest one which is 0.5.032) complains about the Gitlab package location, which stops the deployment.

In the background, we also have created a specific Gitlab user, as prescribed in the admin RStudio Connect guide for private git content:

/etc/rstudio-connect/rstudio-connect.gcfg
[GitCredential]
Host = NOVARTIS-GITLAB
Username = accountName
Password = <encrypted-string>
Protocol = https

Here we are. At that point I don't know what's going on, if it's a packrat issue or an internal issue ...

Thanks for your help

David

[DivadNojnarg]

Here some more details. This is the description of the gitlabrsconnect package:

Package: gitlabrsconnect
Title: What the Package Does (One Line, Title Case)
Version: 0.0.0.9000
Authors@R: 
    person(given = "David",
           family = "Granjon",
           role = c("aut", "cre"),
           email = "dgranjon@ymail.com")
Description: What the package does (one paragraph).
License: `use_mit_license()`, `use_gpl3_license()` or friends to pick a
        license
Encoding: UTF-8
URL: https://NOVARTIS-GITLAB/user/gitlabrsconnect
LazyData: true
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.1.1.9000
RemoteType: gitlab
RemoteHost: NOVARTIS-GITLAB
RemoteRepo: gitlabrsconnect
RemoteUsername: user
RemoteRef: HEAD
RemoteSha: 82570bb00f6d608f7230ac60ae5b94eef14495e4
NeedsCompilation: no
Packaged: 2021-03-23 14:04:41 UTC; user
Author: David Granjon [aut, cre]
Maintainer: David Granjon <dgranjon@ymail.com>
Built: R 3.6.1; ; 2021-03-23 14:04:42 UTC; unix

Here is the rsconnect package description installed from Github, to compare:

Package: rsconnect
Type: Package
Title: Deployment Interface for R Markdown Documents and Shiny
        Applications
Version: 0.8.16-9002
Author: JJ Allaire
Maintainer: Jonathan McPherson <jonathan@rstudio.com>
Description: Programmatic deployment interface for 'RPubs', 'shinyapps.io', and
    'RStudio Connect'. Supported content types include R Markdown documents,
    Shiny applications, Plumber APIs, plots, and static web content.
Depends: R (>= 3.0.0)
Imports: curl, digest, jsonlite, openssl, packrat (>= 0.5), rstudioapi
        (>= 0.5), yaml (>= 2.1.5)
Suggests: RCurl, callr, httpuv, knitr, plumber (>= 0.3.2), reticulate,
        rmarkdown (>= 1.1), shiny, sourcetools, testthat, xtable
License: GPL-2
RoxygenNote: 7.0.2
Roxygen: list(markdown = TRUE)
URL: https://github.com/rstudio/rsconnect
BugReports: https://github.com/rstudio/rsconnect/issues
RemoteType: github
RemoteHost: api.github.com
RemoteRepo: rsconnect
RemoteUsername: rstudio
RemoteRef: master
RemoteSha: 8c11a55bbbe111a4df1e37ddaeb6f19a86f32d39
GithubRepo: rsconnect
GithubUsername: rstudio
GithubRef: master
GithubSHA1: 8c11a55bbbe111a4df1e37ddaeb6f19a86f32d39
NeedsCompilation: no
Packaged: 2021-02-25 10:54:53 UTC; user
Built: R 3.6.1; ; 2021-02-25 10:54:57 UTC; unix

I see 4 extra fields:

GithubRepo: rsconnect
GithubUsername: rstudio
GithubRef: master
GithubSHA1: 8c11a55bbbe111a4df1e37ddaeb6f19a86f32d39

I don't see any Gitlab equivalent, but it seems to be only for backward compatibility with packrat: https://github.com/r-lib/remotes/blob/master/R/install-github.R#L142

[cderv]

Following a discussion with @DivadNojnarg in private, I understand better the issue and I believe this is an issue with rsconnect and not packrat.

rsconnect throws the above message

Unable to determine the location for some packages.

when no repository containing the package is found. This is checked by rsconnect using Repository field in DESCRIPTION. You don't have this field when packages are installed from source. rsconnect check this using rsconnect:::isSCMSource and currently it does not know gitlab

# Return TRUE when the source indicates that a package was installed from
# source or comes from a source control system. This indicates that we will
# not have a repostory URL; location is recorded elsewhere.
isSCMSource <- function(source) {
  tolower(source) %in% c("github", "bitbucket", "source")
}

Source: https://github.com/rstudio/rsconnect/blob/833fb27ddfad149d94877b8e6b0f8d70ef8ab339/R/dependencies.R#L161-L163

So rsconnect must be taught about Gitlab as I don't think if fully knows about it yet, following the work done in packrat (https://github.com/rstudio/packrat/pull/562)

[cderv]

@DivadNojnarg, regarding the new error https://github.com/rstudio/rsconnect/pull/493#issuecomment-807358856, it seems like at an error during installation on the RStudio connect server. It means the manifest has been created and the bundle has been deployed. You should be able to download the source bundle from the app management page, and look into to see if everything looks ok. Maybe there is an information missing in the Manifest and gitlab support in rsconnect is not yet ok.

You can also follow the tips I sent you on Slack grrr to generate the Manifest file and bundles locally.

[cderv]

As side notes on this

We (at Novartis) are currently trying to enable RStudio Connect users to deploy apps depending on private Gitlab packages. Currently, we face an issue.

This is possible that private gitlab is now the issue

In the background, we also have created a specific Gitlab user, as prescribed in the admin RStudio Connect guide for private git content:

Not sure if setting the configuration for git backed content is also applying for restoring packages with Packrat. Seems like packrat works other ways with environment variable (looking at gitlab.R content here) - don't know and not sure if this is compatible with RStudio connect yet.

[DivadNojnarg]

Edit: it appears to be a missing https in the host parameter during the call to remotes::install_gitlab(). This element was missing from the gitlabrsconnect DESCRIPTION file. This was messing with internal functions like packrat:::gitlabDownloadImpl.

Adding this works locally with a packrat::restore() (packrat 0.5.0.32) but still fails on the server (packrat 0.5.0.26) with a nicer user message:

Installing gitlabrsconnect (0.0.0.9000) ... 
[Connect] 2021/03/26 09:05:18.938369110 FAILED
[Connect] 2021/03/26 09:05:18.939070164 Error in getSourceForPkgRecord(pkgRecord, srcDir(project), availablePackagesSource(repos = repos), : Failed to download package from URL:
[Connect] 2021/03/26 09:05:18.939087523 - 'https://<NOVARTIS-GITLAB>/api/v4/projects/user%2Fgitlabrsconnect/repository/archive?sha=82570bb00f6d608f7230ac60ae5b94eef14495e4'
[Connect] 2021/03/26 09:05:18.939118578 

Don't know whether the server packrat version has an impact here.

[DivadNojnarg]

I'll probably open an issue in remotes as the help is slightly misleading:

host GitLab API host to use. Override with your GitLab enterprise hostname, for example, "gitlab.hostname.com".

gitlab.hostname.com is not enough as the protocole would be missing for the downstream packrat functions.

[aronatkins]

From the discussion at https://github.com/rstudio/rsconnect/pull/493, it feels like this is resolved with rsconnect-only changes.

[kevinushey]

Thanks everyone!

[DivadNojnarg]

Thanks!