data.table Reverse Dependency (revdep) Checks

This is code for checking all reverse dependencies of R package data.table. It is designed to be run in the context of a [[https://slurm.schedmd.com/][SLURM]] cluster such as [[https://in.nau.edu/arc/details/][Monsoon]] at Northern Arizona University. Results are computed every day, and results for the past few days are available for inspection in the [[https://rcdata.nau.edu/genomic-ml/data.table-revdeps/analyze/][analyze]] directory.

** Overview of scripts

[[file:params.R]] launches the revdep checks, by first building data.table from github master, and optionally rebuilding R itself, then

• using sbatch with a job array, each task for a different revdep to check via [[file:check_one.R]]
• using sbatch with ~--depend=afterany:JOBID~ to run [[file:analyze.R]] to make the [[https://rcdata.nau.edu/genomic-ml/data.table-revdeps/analyze/][output web page]] after all the checks finish.

So that the job is run every day, there are two options

• create a crontab entry which runs [[file:sbatch_params.sh][sbatch_params.sh]], which will ask the cluster to run params.sh, which is generated in https://github.com/tdhock/data.table-revdeps/blob/master/params.R#L319 (this is what we do currently).
• using sbatch with ~--begin=tomorrow~ at 00:01 to run [[file:params.R]] again, which will start the checks tomorrow (on a compute node, not done currently because we use the cron option above instead).

cron runs params.sh with a minimal environment which can be simulated for testing purposes via

+BEGIN_SRC shell-script

env -i bash --noprofile --norc ~/bin/params.sh

+END_SRC

** Software required

The scripts are currently highly specific to data.table and to the NAU Monsoon cluster. System libraries required to build R packages are installed in my [[file:emacs1-env-from-history.yml][conda environment]] and/or home directory:

+BEGIN_SRC

th798@wind:~/src$ ls QuantLib-1.28 icu openssl-1.1.1s rasqal-0.9.33 SYMPHONY-5.6.17 icu4c-70_1-src.tgz raptor2-2.0.15 redland-1.0.17

+END_SRC

ICU 70 is installed under $HOME (for redland), and ICU 58 is installed under $CONDA_PREFIX (for base R).

** Testing status

This software is highly experimental and has not been extensively tested.

** Related work

• [[https://github.com/Rdatatable/data.table/blob/master/.dev/revdep.R]] implements revdep checking for data.table on your local computer.
• [[https://github.com/r-lib/revdepcheck]] implements revdep checking for any R package (not just data.table), and implements parallel processing for RStudio employees in the AWS cloud.