using source-overlay errs on second restore

[aronatkins]

This issue shows that packrat can restore from bundles in a source overlay directory but is unable to use those installed packages on subsequent restore. We are seeing this problem while using the packrat cache, but the steps here show that the cache is not necessary.

• Create a toy package that we will "pretend" is on GitHub.
• Create a tar.gz bundle using R CMD build completelyPrivatePackage.
• Add that tar.gz into a "sources" directory with a fake-SHA filename.
• Create a toy project that uses this package.
• Restore that project (using the sources directory).
• Restore that project again.

example `packrat.lock`

``` PackratFormat: 1.4 PackratVersion: 0.4.4 RVersion: 3.2.0 Repos: CRAN=http://cran.rstudio.com Package: completelyPrivatePackage Source: github Version: 0.1.0 Hash: 88e8036bdbc415864d0f021ce6017997 GithubRepo: completelyPrivatePackage GithubUsername: this-user-does-not-exist GithubRef: master GithubSha1: decafbaddecafbaddecafbaddecafbaddecafbad RemoteHost: api.github.com RemoteRepo: completelyPrivatePackage RemoteUsername: this-user-does-not-exist RemoteRef: master RemoteSha: decafbaddecafbaddecafbaddecafbaddecafbad ```

In my toy project, I have a source-overlay directory that contains:

source-overlay/completelyPrivatePackage/decafbaddecafbaddecafbaddecafbaddecafbad.tar.gz

Here is a restore script that uses this overlay directory:

sourceOverlay <- file.path(getwd(), "source-overlay")
Sys.setenv(R_PACKRAT_SRC_OVERLAY = sourceOverlay)
Sys.setenv(R_PACKRAT_SRC_DIR = file.path(tempdir(), "packrat-src"))

options(packrat.untrusted.packages = character())
options(
  packrat.verbose.cache = TRUE,
  packrat.connect.timeout = 10
)

packrat::set_opts(
  auto.snapshot = FALSE,
  project = getwd(),
  persist = FALSE
)

packrat::restore(
  overwrite.dirty = TRUE,
  prompt = FALSE,
  restart = FALSE
)

On first run (no packrat/lib):

R --vanilla -s -f ./doit.R
Installing completelyPrivatePackage (0.1.0) ...
    OK (built source)
Warning messages:
1: In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0
2: In untar(src, exdir = target, compressed = "gzip") :
  argument 'compressed' is ignored for the internal method
3: In untar(src, compressed = "gzip", list = TRUE) :
  argument 'compressed' is ignored for the internal method

On second run (after that successful restore):

R --vanilla -s -f ./doit.R
Error: Unable to retrieve package records for the following packages:
- 'completelyPrivatePackage'
In addition: Warning message:
In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0
Execution halted

The restore works by adding the following fields to the DESCRIPTION file within that sources tar.gz file:

Source: github
GithubRepo: completelyPrivatePackage
GithubUsername: this-user-does-not-exist
GithubRef: master
GithubSHA1: decafbaddecafbaddecafbaddecafbaddecafbad
RemoteHost: api.github.com
RemoteRepo: completelyPrivatePackage
RemoteUsername: this-user-does-not-exist
RemoteRef: master
RemoteSha: decafbaddecafbaddecafbaddecafbaddecafbad

With this additional metadata, the second restore succeeds:

R --vanilla -s -f ./doit.R
Already up to date.
Warning message:
In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0

Note: You'll need to remove the packrat/lib when switching between the original and augmented source bundles.

It feels as if packrat is seeing the presence of a tar.gz in the sources directory: https://github.com/rstudio/packrat/blob/448aafd176ec6cbde307f556c91b39e5a1c94f9d/R/restore.R#L91-L92 That prevents us from adding DESCRIPTION fields associated with GitHub packages: https://github.com/rstudio/packrat/blob/448aafd176ec6cbde307f556c91b39e5a1c94f9d/R/restore.R#L307-L328

We add fields for other types of repositories (bitbucket, gitlab, etc). Packages from these sources probably have the same issue.

[aronatkins]

Packrat probably should not touch bundles in the source overlay directory, instead modifying them on-the-fly as it prepares to install from that file.