Review of scripts and env. variables provided by members of CRAN

[statnmap]

Context

In August 2022, we received some links pointing to resources run by the CRAN team to (1) check incoming tar.gz packages, (2) regularly check all packages on CRAN.
The answers are there: https://github.com/RConsortium/r-repositories-wg/blob/main/Documents/Proposal%20to%20CRAN.md

In this issue, we can discuss the test of the scripts received:

• What do we learn ?
• What is needed to make them work ?
• How are they usable on different OS flavours ?

Scripts

Let's start with the incoming check as there are the first step to pass to go to CRAN.
Then, we may explore regular checks.

Incoming checks

• Incoming checks are run each time there is a new packages send to CRAN

=> It seems that each member of CRAN runs a different set of test, depending on their OS

Scripts are:

• A Rscript that prepares the system and run the 'R CMD check" command: https://github.com/r-devel/r-dev-web/blob/master/CRAN/QA/Kurt/lib/R/Scripts/check_CRAN_incoming.R
• It is amended with env. variables: https://github.com/r-devel/r-dev-web/blob/master/CRAN/QA/Kurt/.R/check.Renviron

=> It seems that they do not use the --as-cran tag to run the check.

:heavy_check_mark: Tests on Ubuntu 22.04 LTS

Extra steps needed on local computer:

mkdir ~/tmp/
mkdir ~/tmp/scratch
mkdir ~/tmp/CRAN

• System function getIncoming is used in the R file but does not exist on my system.

=> Need to comment the getIncoming part (L173-182) => Put the tar.gz of your package inside the check_dir, which by default is "~/tmp/CRAN" => Run the R code => The output looks like what we receive by email

Depends:
Package: fusen
  Depends: R (>= 3.5.0)
  Imports: attachment, cli, desc, devtools, glue, here (>= 1.0.0),
    magrittr, parsermd (>= 0.1.0), roxygen2, stats, stringi,
    tibble, tidyr, tools, usethis (>= 2.0.0), utils

Timings:
      utilisateur système  écoulé
fusen     107.585   9.573 143.396

Results:
Check status summary:
                  ERROR
  Source packages     1

Check results summary:
fusen ... ERROR
* checking CRAN incoming feasibility ... NOTE
* checking tests ... ERROR
* checking PDF version of manual ... WARNING

The full check directory is stored in the check_dir, which by default is "~/tmp/CRAN"

:heavy_check_mark: it works on Ubuntu 20.04 LTS as is

Regular checks

• Regular checks seems to be run regularly.
• Question is when ? when the devel version of R change, dependencies, nightly, .... ?
• These checks seem to update the check result page of each package
  • e.g. https://cran.r-project.org/web/checks/check_results_gitlabr.html

Scripts are:

• sh script: https://github.com/r-devel/r-dev-web/blob/master/CRAN/QA/Kurt/bin/check-R-ng

=> This builds the R-devel version

• It runs a R script: https://github.com/r-devel/r-dev-web/blob/master/CRAN/QA/Kurt/lib/R/Scripts/check_CRAN_regular.R
• Which itself uses this list of env. variables: https://github.com/r-devel/r-dev-web/blob/master/CRAN/QA/Kurt/.R/check.Renviron

=> This builds all R base packages

Tests on Ubuntu 22.04 LTS

• Extra steps needed on local computer:

• mkdir ~/tmp/
  mkdir ~/tmp/R.check

• Define these two directories inside file "check-R-ng"

• R scripts directory :

  R_scripts_dir=~/lib/R/Scripts

• Shell scripts directory.

  sh_scripts_dir=~/lib/bash

• Error to be explored....

  creating NEWS.2.pdf
  /bin/bash: line 1: html: command not found
  make: [Makefile:120: R-FAQ.epub] Error 127 (ignored)
  /usr/bin/sed: can't read R-FAQ.epub.tmp: No such file or directory
  make: *** [Makefile:121: R-FAQ.epub] Error 2
  Fatal error: cannot open file '/home/srochett/lib/R/Scripts/check_CRAN_regular.R': No such file or directory

=> Need to be tested inside a Docker container

[statnmap]

I added what I could in {checkhelper} for tests

Experimental: Check as CRAN with CRAN global variables

Use the exploration of CRAN scripts by the RConsortium to check a package as CRAN does it with their env. variables. See https://github.com/RConsortium/r-repositories-wg/issues/17 for more details.

install.packages('checkhelper', repos = 'https://thinkr-open.r-universe.dev')

check_dir <- tempfile("example")
# Check the current directory
check_as_cran(check_dir = check_dir)
# Open directory with all outputs
utils::browseURL(check_dir)

[hadley]

Thanks @statnmap! Would you mind tweaking your links to use https://github.com/r-devel/r-dev-web? That's a bit easier to navigate interactively.

[hadley]

I started putting together a brief big picture overview at https://docs.google.com/document/d/1H41Jjuj6T7o7zdhe4mh2rPyIi76YkpnY4Jd48tBBy_Q/edit?usp=sharing

[statnmap]

• I changed the name for the github repo
• Note that I started to put this information in my PR: https://github.com/RConsortium/r-repositories-wg/pull/18/files#diff-4544a1c79c6dc79452cf47adb2cb75f54f8b62b67eacfcc13547267337f059ab Don't we want to keep track of our exploration in the present repository, instead of an external file ?

Also, I was wondering. I think that Gabor already explored all these files to build {rhub} containers like CRAN. Maybe we do not want to do the job a second time, and better have a look at what changed between the last time Gabor explored CRAN files and now ?

[llrs]

As already mention I don't find clear when are the regular checks run:

I know the incoming tests are not performed all at the same time: windows checks are done later after a package is already accepted to CRAN. That's why sometimes maintainers get a message accepting their package and shortly after a notice to update their package to keep them in CRAN.

However, how does this happens for packages that are already in CRAN? How/when are the test run? Presumably there is some cron jobs (or similar) that would start checks or the scripts will decide which packages to test for instance based on change in dependencies, or updates in r-devel or machine changes. However, this doesn't seem to be happening as often as I expected (daily in all setups), and I'm not sure where in the script this decisions are made.

Let me give an example (Perhaps it should be also added to #20 ) : lifecycle 1.0.3 has recently (2022-10-07) updated the messages output. The package is already in CRAN so presumably the reverse check dependencies were successfully. rtweet (which I maintain) uses it and has snapshots tests with the old output, so now checks should fail as I found out this via github checks. However, after 3 days none of the checks display any ERROR about said change (the webpage says it was updated today 2022-10-10 08:49:29 CEST).

Looking at the individual check pages of rtweet, at the moment of posting this says:

Flavor	First line
r-devel-linux-x86_64-debian-clang	2022-10-04 r83023
r-devel-linux-x86_64-debian-gcc	2022-10-08 r83047
r-devel-linux-x86_64-fedora-clang	2022-10-07 r83044
r-devel-linux-x86_64-fedora-gcc	2022-10-07 r83044
r-devel-windows-x86_64	2022-10-07 r83044 ucrt
r-patched-linux-x86_64	2022-10-03 r83013
r-release-linux-x86_64	4.2.1 (2022-06-23)
r-release-macos-arm64	4.2.0 (2022-04-22)
r-release-macos-x86_64	4.2.0 (2022-04-22)
r-release-windows-x86_64	4.2.1 (2022-06-23 ucrt)
r-oldrel-macos-arm64	4.1.1 Patched (2021-09-05 r80862)
r-oldrel-macos-x86_64	4.1.2 (2021-11-01)
r-oldrel-windows-ix86+x86_64	4.1.3 (2022-03-10)

It is not clear to me when test were started for r-release, r-oldrel and r-patched version, and even with r-devel there are checks that haven't been run in a couple of days.

[hadley]

@llrs snapshot tests aren't run on CRAN