Packrat does not download from a private repository on github

[leidos-cap]

I have a private repository on GitHub while the R library is in development. I run R and install the package from the command line without a problem: devtools::install_github(repo = "leidos-cap/p-medds", subdir = "pmedds.core", auth_token = my github token)

With this command in the code and when packrat tries it the following messages are returned: curl: (22) The requested URL returned error: 404 Not Found and Errors occurred when fetching source files: Error in getSourceForPkgRecord(pkgRecord, sourceDir, availablePkgs, repos) : Failed to download package from URL: -'https://www.github.com/leidos-cap/p-medds/archive/cb114128be6c11c8115785db48e8d73dd4649690.tar.gz'

Any ideas on how to get this to work or does packrat not download priviate repositories.

Thanks

[kevinushey]

What version of packrat are you running? The latest version of packrat should be able to handle this, as long as you have the devtools package installed and available in your project.

[leidos-cap]

Packrat 0.4.7-1 is installed.

The start of server.R is: rm(list=ls()) library(devtools) library(packrat) library(shiny) library(dplyr) library(coda) library(fields) library(ggplot2) library(gtable) library(grid) library(gridExtra) library(scales) library(shinyFiles) library(utils) if (! require("rsconnect")) { devtools::install_github("rstudio/rsconnect") library ("rsconnect") } if( ! require("pmedds.core")) { devtools::install_github(repo = "leidos-cap/p-medds", subdir = "pmedds.core", auth_token = "auth-code") library(pmedds.core) }

running packrat returns

packrat::init() Initializing packrat project in directory:

• "/u2/workspace/nextgen/P-MEDDS/pmeddsGraphicApp/R"

Adding these packages to packrat: _
BH 1.60.0-1 DBI 0.3.1 PKI 0.1-3 R6 2.1.2 RColorBrewer 1.1-2 RCurl 1.95-4.8 RJSONIO 1.3-0 RSQLite 1.0.0 Rcpp 0.12.4 assertthat 0.1 base64enc 0.1-3 bitops 1.0-6 coda 0.18-1 colorspace 1.2-6 curl 0.9.6 devtools 1.10.0 dichromat 2.0-0 digest 0.6.9 dplyr 0.4.3 fields 8.3-6 ggplot2 2.1.0 git2r 0.13.1 gridExtra 2.2.1 gtable 0.2.0 htmltools 0.3.5 httpuv 1.3.3 httr 1.1.0 jsonlite 0.9.19 labeling 0.3 lazyeval 0.1.10 magrittr 1.5 maps 3.1.0 memoise 1.0.0 mime 0.4 munsell 0.4.3 openssl 0.9.2 packrat 0.4.7-1 plyr 1.8.3 pmedds.core 1.0 reshape2 1.4.1 rsconnect 0.4.2.2 rstudioapi 0.5 scales 0.4.0 shiny 0.13.2 shinyFiles 0.6.0 spam 1.3-0 stringi 1.0-1 stringr 1.0.0 whisker 0.3-2 withr 1.0.1 xtable 1.8-2 yaml 2.1.13

Fetching sources for BH (1.60.0-1) ... OK (CRAN current) Fetching sources for DBI (0.3.1) ... OK (CRAN current) Fetching sources for PKI (0.1-3) ... OK (CRAN current) Fetching sources for R6 (2.1.2) ... OK (CRAN current) Fetching sources for RColorBrewer (1.1-2) ... OK (CRAN current) Fetching sources for RCurl (1.95-4.8) ... OK (CRAN current) Fetching sources for RJSONIO (1.3-0) ... OK (CRAN current) Fetching sources for RSQLite (1.0.0) ... OK (CRAN current) Fetching sources for Rcpp (0.12.4) ... OK (CRAN current) Fetching sources for assertthat (0.1) ... OK (CRAN current) Fetching sources for base64enc (0.1-3) ... OK (CRAN current) Fetching sources for bitops (1.0-6) ... OK (CRAN current) Fetching sources for coda (0.18-1) ... OK (CRAN current) Fetching sources for colorspace (1.2-6) ... OK (CRAN current) Fetching sources for curl (0.9.6) ... OK (CRAN archived) Fetching sources for devtools (1.10.0) ... OK (CRAN current) Fetching sources for dichromat (2.0-0) ... OK (CRAN current) Fetching sources for digest (0.6.9) ... OK (CRAN current) Fetching sources for dplyr (0.4.3) ... OK (CRAN current) Fetching sources for fields (8.3-6) ... OK (CRAN current) Fetching sources for ggplot2 (2.1.0) ... OK (CRAN current) Fetching sources for git2r (0.13.1) ... OK (CRAN archived) Fetching sources for gridExtra (2.2.1) ... OK (CRAN current) Fetching sources for gtable (0.2.0) ... OK (CRAN current) Fetching sources for htmltools (0.3.5) ... OK (CRAN current) Fetching sources for httpuv (1.3.3) ... OK (CRAN current) Fetching sources for httr (1.1.0) ... OK (CRAN current) Fetching sources for jsonlite (0.9.19) ... OK (CRAN current) Fetching sources for labeling (0.3) ... OK (CRAN current) Fetching sources for lazyeval (0.1.10) ... OK (CRAN current) Fetching sources for magrittr (1.5) ... OK (CRAN current) Fetching sources for maps (3.1.0) ... OK (CRAN current) Fetching sources for memoise (1.0.0) ... OK (CRAN current) Fetching sources for mime (0.4) ... OK (CRAN current) Fetching sources for munsell (0.4.3) ... OK (CRAN current) Fetching sources for openssl (0.9.2) ... OK (CRAN current) Fetching sources for packrat (0.4.7-1) ... OK (CRAN current) Fetching sources for plyr (1.8.3) ... OK (CRAN current) Fetching sources for pmedds.core (1.0) ... curl: (22) The requested URL returned error: 404 Not Found curl: (22) The requested URL returned error: 404 Not Found curl: (22) The requested URL returned error: 404 Not Found curl: (22) The requested URL returned error: 404 Not Found curl: (22) The requested URL returned error: 404 Not Found FAILED Fetching sources for reshape2 (1.4.1) ... OK (CRAN current) Fetching sources for rsconnect (0.4.2.2) ... OK (GitHub) Fetching sources for rstudioapi (0.5) ... OK (CRAN current) Fetching sources for scales (0.4.0) ... OK (CRAN current) Fetching sources for shiny (0.13.2) ... OK (CRAN current) Fetching sources for shinyFiles (0.6.0) ... OK (CRAN current) Fetching sources for spam (1.3-0) ... OK (CRAN current) Fetching sources for stringi (1.0-1) ... OK (CRAN current) Fetching sources for stringr (1.0.0) ... OK (CRAN current) Fetching sources for whisker (0.3-2) ... OK (CRAN current) Fetching sources for withr (1.0.1) ... OK (CRAN current) Fetching sources for xtable (1.8-2) ... OK (CRAN current) Fetching sources for yaml (2.1.13) ... OK (CRAN current) Error in snapshotSources(project, activeRepos(project), allRecordsFlat) : Errors occurred when fetching source files: Error in getSourceForPkgRecord(pkgRecord, sourceDir, availablePkgs, repos) : Failed to download package from URL:

• 'https://www.github.com/leidos-cap/p-medds/archive/cb114128be6c11c8115785db48e8d73dd4649690.tar.gz'

When running install_github from command prompt:: devtools::install_github(repo = "leidos-cap/p-medds", subdir = "pmedds.core", auth_token = "auth_token")

Downloading GitHub repo leidos-cap/p-medds@master from URL https://api.github.com/repos/leidos-cap/p-medds/zipball/master ...... The library installs.

I notice that the download URL is different from when packrat tries.

[leidos-cap]

I just download the packrat repository from GitHub and found the code for installing packages from GiHub:

Install packages from GitHub

if (length(githubPkgs)) { message("> Installing GitHub packages") if (!requireNamespace("devtools")) { install.packages("devtools", lib = userLib()) } for (pkg in githubPkgs) { desc <- as.data.frame(descContent[[pkg]], stringsAsFactors = FALSE) ref <- desc$GithubSHA1 %||% desc$GithubRef %||% "master" devtools::install_github(repo = desc$GithubRepo, username = desc$GithubUsername, ref = ref, quick = TRUE ) } }

It does not have the auth_token argument!!

I thought to use Sys.setenv(GITHUB_PAT="auth_token") and added this before any libraries in server.R. This did not work.

If I either use Sys.setenv() from the R command line or set a system GETHUB_PAT variable before running packrat, it worked.

Unfortunately the website I am going to be running this code on requires that the code be submitted as a zip file, the website installs the code and runs packrat. There is not currently a way to set GITHUB_PAT before packrat is run.

Any ideas?

[kevinushey]

That's not the code that installs GitHub packages. That's old migration code that's not used in that particular code path.

All the relevant downloader code is here: https://github.com/rstudio/packrat/blob/master/R/github.R

It's called within the downloader code here: https://github.com/rstudio/packrat/blob/master/R/downloader.R#L38-L44

Is devtools installed? Can you try installing it explicitly first, then calling packrat::snapshot()?

[leidos-cap]

devtools is installed. Both in the local repository and in packrat (if pmedds.core installed).

I run packrat::init, with GITHUBPAT set and then ran packrat::snapshot. The result was The following packages are stale:
pmedds.core 1.0

These packages must be updated by calling packrat::restore() before snapshotting. If you are sure you want the installed versions of these packages to be snapshotted, call packrat::snapshot() again with ignore.stale=TRUE.

ran packrat::restore() then packrat::snapshot() again. Got the same results.

[leidos-cap]

I think I see the problem. The githubDownload function uses the github_pat function which uses the system environment variable. The install_github argument auth_token is ignored as far as I can tell.

This seems to be a bug, both GITHUB_PAT and the install_github auth_token should work.

[kevinushey]

Strange -- the signature of install_github() suggests that it uses the github_pat() by default, e.g.

> devtools::install_github
function (repo, username = NULL, ref = "master", subdir = NULL, 
    auth_token = github_pat(quiet), host = "api.github.com", 
    force = FALSE, quiet = FALSE, ...) 
{
    remotes <- lapply(repo, github_remote, username = username, 
        ref = ref, subdir = subdir, auth_token = auth_token, 
        host = host)
    if (!isTRUE(force)) {
        remotes <- Filter(function(x) different_sha(x, quiet = quiet), 
            remotes)
    }
    install_remotes(remotes, quiet = quiet, ...)
}
<environment: namespace:devtools>

And the githubDownload() function should explicitly use httr to pass along that auth token.

If you call debug(packrat:::devtoolsDownload), does that ever get called for your GitHub packages? What's the output when you try to step through the relevant calls to httr?

It would also be good to check that packrat:::canUseGitHubDownloader() returns TRUE.

[leidos-cap]

The GITHUB_PAT env variable does work with packrat, but I have no way of setting an environmental variable on the system to which this system is being deployed.

From what I saw in the packrat function githubDownload(), install_github() is not used, but GET(url, auth) is used to download the file, where auth is set to the return of github_pat(), which returns the GITHUB_PAT env variable.

I am assuming, because I have not checked, that the install_github() arguments are passed to the packrat download() function, which then passes the arguments to the packrat githubDownload(). If the githubDownload() function checked for an auth_token argument and, if present, use it instead of the github_pat() function to set the auth value of GET.

See the links four comments above to seen the code for download and githubDownload.

[kevinushey]

Okay, I think I understand now -- the overarching issue is that you don't have a mechanism for setting the GITHUB_PAT environment variable on the deployment machine (which packrat uses for authentication), but it does work locally. Is that correct?

Indeed, the only mechanism Packrat provides for authentication with private GitHub repositories is through the GITHUB_PAT environment variable; packrat itself doesn't provide a separate mechanism for passing along this auth token for deployment scenarios.

Note that devtools::install_github() itself is never actually used by packrat when snapshotting a project; packrat constructs the relevant URL itself based on the package's DESCRIPTION file and that is downloaded through packrat:::download().

[leidos-cap]

I assume that packrat has to read the R code and pick out the libraries required to run the application. Is there a way for packrat to find a Sys.setenv(GITHUB_PAT="...") and run it?

I think that packrat not being able to use the auth_token value from install_github is a bug.

[kevinushey]

I'd definitely be open to adding that. Some options:

Make it a project option, so that you could write something like:

packrat::opts$github.pat("<token>")

The PAT would then be persisted as part of packrat/packrat.opts, packrat could read + set that on initialization.

Alternatively, packrat could read a dotfile in the project's folder, e.g. look for a file called .github-pat, and set the GITHUB_PAT environment variable based on its contents (again, on initialization).

Or (if we didn't want the PAT to live with a particular packrat project) we could look for it in a separate location, e.g. ~/.github-pat.

Any thoughts on which is preferable?

[leidos-cap]

Thanks, I will try these.

That is packrat::opts$github.pat and not packrat::opts$github_pat ?

[kevinushey]

Note that none of these are available yet, these are just proposals. Any thoughts on what would work best for your deployment scenario?

[leidos-cap]

Either would work.

The file would be best if the library code would be made public at some time. This way you only have to remove a file and not edit the code.

[leidos-cap]

Thanks for the help. I look forward to the update.

[ElianoMarques]

@kevinushey what was the solution for this? Did you add anything to packrat that makes him download private repos?

[kevinushey]

Yes, packrat should now try to use devtools when downloading from private repositories during restore, assuming devtools is available in that project.

[ElianoMarques]

And this would be the way to use with rstudio connect? I tried the other day and it didn't work with a private repo, even if I installed via devtools::install_github.

[kevinushey]

Unfortunately, this won't work yet with RStudio connect as there is no mechanism for transferring the necessary credentials to the connect server (ie, for setting the GITHUB_PAT token). You'll have to talk to the connect team about workarounds for handling that.

[DataStrategist]

sorry for issue rezing, but I have recently run into this issue. I explicitly called library(devtools) and then tried to packrat::init() but got the same github error mentioned above. Using packrat v 0.4.8-01