E2E tests are flaky for macOS

[kamilzyla]

Our e2e-test.yml workflow fails randomly on macOS. Rerunning typically helps, but sometimes needs to be done more than once. It seems that the failure always happens in the following line:

rhino::pkg_install("Appsilon/shiny.i18n")

of the test-dependencies.R file which is run by the pkg_instal()... step.

[kamilzyla]

I performed extensive testing using a bare-bones workflow on an orphan branch and found out that:

1. The problem seems to be independent of Rhino and our E2E setup. It boils down to renv::install() failing randomly on macOS when installing a package from GitHub, e.g. renv::install("r-lib/rlang").
2. Setting GITHUB_PAT for the step seems to resolve the issue.

I didn't exactly get to the root cause but I don't have time to dig deeper. See below for test code and results.

debug-flaky-e2e.yml

name: Debug Flaky E2E
on: [push]
permissions:
  contents: read
jobs:
  main:
    name: ${{ matrix.id == 0 && 'ubuntu' || format('macOS {0}', matrix.id) }}
    runs-on: ${{ matrix.id == 0 && 'ubuntu-latest' || 'macos-latest' }}
    timeout-minutes: 10
    strategy:
      fail-fast: false
      matrix:
        id: [0, 1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: r-lib/actions/setup-r@v2
      - uses: r-lib/actions/setup-r-dependencies@v2
        with:
          dependencies: '"hard"'
          packages: curl remotes renv withr
      - run: Rscript test.R
        env:
          GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}

test.R

retry <- function(name, f, n = 5, delay = 3) {
  for (i in 1:n) {
    ok <- tryCatch(
      {
        f()
        TRUE
      },
      error = function(e) {
        message("error: ", conditionMessage(e))
        FALSE
      }
    )
    if (ok) break
    else Sys.sleep(delay)
  }
  paste(name, i, ifelse(ok, "OKAY", "FAIL"), sep = "-")
}

PKG <- "r-lib/rlang"
URL <- "https://api.github.com/repos/r-lib/rlang/contents/DESCRIPTION"

results <- character(0)

cat("::group::download\n")
withr::with_tempdir({
  results <- c(results, retry("download", function() download.file(URL, "DESCRIPTION")))
})
cat("::endgroup::\n")

cat("::group::curl\n")
withr::with_tempdir({
  results <- c(results, retry("curl", function() curl::curl_download(URL, "DESCRIPTION")))
})
cat("::endgroup::\n")

cat("::group::remotes\n")
results <- c(results, retry("remotes", function() remotes::install_github(PKG)))
cat("::endgroup::\n")

# Run renv in the end as it modifies the library paths.
cat("::group::renv\n")
withr::with_tempdir({
  renv::init()
  results <- c(results, retry("renv", function() renv::install(PKG)))
})
cat("::endgroup::\n")

cat("::notice::", results, "\n")

Sample run without GITHUB_PAT

Note the renv-5-FAIL (made 5 renv::install() attempts without success) for macOS 4.

Sample run with GITHUB_PAT

The runs were always successful in my experiments.