Make sure r-universe is a compatible source for the package cache

[zkamvar]

François tried out creating a new repository with {sandpaper} and he ran into an interesting problem: because he installed his {renv} package from https://zkamvar.r-universe.dev, there was a problem he ended creating a lockfile that could not be restored because it contained a repository that was not listed at the top:

https://github.com/fmichonneau/test-renv/blob/991dba347bfd65c3e49f49537b982b92d1cee438/renv/profiles/lesson-requirements/renv.lock

He ended up getting this error:

ℹ Consent to use package cache provided
* "~/Library/Application Support/renv" has been created.
→ Searching for and installing available dependencies
* Discovering package dependencies ... Done!
* Copying packages into the cache ... Done!
* Resolving missing dependencies  ... 
Retrieving 'https://cran.rstudio.com/bin/macosx/contrib/4.1/cowsay_0.8.0.tgz' ...
    OK [downloaded 391.9 Kb in 0.1 secs]
Retrieving 'https://cran.rstudio.com/bin/macosx/contrib/4.1/crayon_1.4.1.tgz' ...
    OK [downloaded 136.5 Kb in 0.1 secs]
Retrieving 'https://cran.rstudio.com/bin/macosx/contrib/4.1/fortunes_1.5-4.tgz' ...
    OK [downloaded 204.3 Kb in 0.1 secs]
Retrieving 'https://cran.rstudio.com/bin/macosx/contrib/4.1/rmsfact_0.0.3.tgz' ...
    OK [downloaded 18.5 Kb in 0.1 secs]
Installing crayon [1.4.1] ...
    OK [installed binary]
Moving crayon [1.4.1] into the cache ...
    OK [moved to cache in 1 milliseconds]
Installing fortunes [1.5-4] ...
    OK [installed binary]
Moving fortunes [1.5-4] into the cache ...
    OK [moved to cache in 0.95 milliseconds]
Installing rmsfact [0.0.3] ...
    OK [installed binary]
Moving rmsfact [0.0.3] into the cache ...
    OK [moved to cache in 0.95 milliseconds]
Installing cowsay [0.8.0] ...
    OK [installed binary]
Moving cowsay [0.8.0] into the cache ...
    OK [moved to cache in 0.97 milliseconds]
→ Restoring any dependency versions
The following package(s) will be updated:

# RSPM ===============================
- htmltools   [0.5.2 -> 0.5.2]
- knitr       [1.33 -> 1.33]
- magrittr    [2.0.1 -> 2.0.1]
- rlang       [0.4.11 -> 0.4.11]
- rmarkdown   [2.10 -> 2.10]
- stringi     [1.7.4.9001 -> 1.7.4]
- tinytex     [0.33 -> 0.33]
- xfun        [0.25 -> 0.25]

# https://zkamvar.r-universe.dev =====
- renv        [* -> 0.14.0-8]

* Querying repositories for available binary packages ... Done!
Retrieving 'https://cran.rstudio.com/bin/macosx/contrib/4.1/stringi_1.7.4.tgz' ...
    OK [downloaded 14 Mb in 0.2 secs]
* Querying repositories for available source packages ... Done!
Error : failed to retrieve package 'renv'

Error: Error: <callr_status_error: callr subprocess failed: failed to retrieve package 'renv'>
-->
<callr_remote_error in NULL:
 failed to retrieve package 'renv'>
 in process 4093 

 Stack trace:

 Process 4039:
 1. sandpaper::manage_deps(path = ".", quiet = FALSE)
 2. callr::r(func = callr_manage_deps, args = args, show = !quiet,  ...
 3. callr:::get_result(output = out, options)
 4. throw(newerr, parent = remerr[[2]])

 x callr subprocess failed: failed to retrieve package 'renv' 

 Process 4093:
 16. (function (path, repos, snapshot, lockfile_exists)  ...
 17. renv::restore(library = renv::paths$library(), lockfile = renv::paths$lockf ...
 18. renv:::renv_restore_run_actions(project, diff, current, lockfile,  ...
 19. renv:::retrieve(packages)
 20. renv:::handler(package, renv_retrieve_impl(package))
 21. renv:::renv_retrieve_impl(package)
 22. renv:::renv_retrieve_repos(record)
 23. renv:::stopf("failed to retrieve package '%s'", record$Package)
 24. base:::stop(sprintf(fmt, ...), call. = call.)
 25. base:::.handleSimpleError(function (e)  ...
 26. h(simpleError(msg, call))

 x failed to retrieve package 'renv' 

When he re-installed {renv} from CRAN, it worked:

https://github.com/fmichonneau/test-renv-2/blob/main/renv/profiles/lesson-requirements/renv.lock

[zkamvar]

Another potentially separate issue, but errors might appear at first when trying to hydrate, but resolve themselves later as packages get installed via restore: https://codimd.carpentries.org/_4KFOwW-R0eYgwkhu6SIdg?view

[zkamvar]

Another data point:

https://github.com/zkamvar/test-with-renv/runs/3861848594?check_suite_focus=true

# https://carpentries.r-universe.dev =
- stringi      [*/*@HEAD: dfc45538 -> 7248caee]

* Querying repositories for available binary packages ... Done!
* Querying repositories for available source packages ... Done!
Error : failed to retrieve package 'stringi'

I've added {stringi} to The Carpentries r-universe, but here's the problem:

{renv} expects all versions of a package to be retrievable from repository archives based on the version number. This unfortunately breaks when you have a package taken from an R-Universe. Why you ask? It's because R-universe is not an archive. When stringi updated from 1.7.4.9000 to 1.7.5, the devel version was lost and renv got confused.

Luckily, there may be a solution to this by setting the renv.retrieve.repos.archive.path option: https://github.com/rstudio/renv/issues/602

[zkamvar]

Update: I have attempted to create a solution based on what I thought the option above was doing, but to no avail:

over <- function(repo, record) { 
  if (!grepl('r[-]universe[.]dev.?$', repo)) return(NULL) 
  vsn <- gsub("[^.0-9]", '', suppressWarnings(readLines(paste0(repo, "/packages/", record$Package)))) 
  vsn <- vsn[vsn != ''] 
  url <- contrib.url(repo, type = "binary") 
  sfx <- if (grepl("macos", url)) ".tgz" else if (grepl("windows", url)) ".zip" else ".tar.gz" 
  paste0(url, "/", record$Package, "_", vsn, sfx) 
}

The issue here is that the option above is indeed looking for an archive link that will have that specific version, so we end up with errors like this:

Retrieving 'https://carpentries.r-universe.dev/src/contrib/stringi_1.7.5.tar.gz/stringi_1.7.4.9001.tar.gz' ...
Retrieving 'https://cran.rstudio.com/src/contrib/Archive/stringi/stringi_1.7.4.9001.tar.gz' ...
Error: failed to retrieve package 'stringi'
In addition: Warning messages:
1: curl: (22) The requested URL returned error: 404 Not Found 
2: curl: (22) The requested URL returned error: 404 Not Found 

[zkamvar]

The solution here is to use sandpaper::pin_version() and rebuild, but that's not really all that satisfying.