Get `renv` to work on HPC

MiguelRodo commented 10 hours ago

Site-wide .Renviron file

RENV_PATHS_ROOT=/renv/local
RENV_PATHS_LIBRARY_ROOT=/workspaces/.local/lib/R/library
RENV_PATHS_CACHE=/renv/cache
R_LIBS=/workspaces/.local/lib/R

When you start R on the HPC, these are active

MiguelRodo commented 10 hours ago

Trying to run `apptainer` in writable mode

So, the /workspaces folder is read only. This might be a problem. Should be just run apptainer in like editable mode?

Two ways of doing this:
- Writable sandbox
- This does not work straight away, as /scratch is not mounted.
- I did the following:
  - Created the sandbox: apptainer build --sandbox comp24stimgate/ comp24stimgate.sif
  - Ran with the sandbox: apptainer shell --writable comp24stimgate/
  - Got this warning: Skipping mount /scratch [binds]: /scratch doesn't exist in container
  - When in the container, running ls / shows that scratch is definitely not there.
- Overlay file system
- I tried this, but it's very slow for some reason in R, unless I run R --vanilla. I don't know why...
- For future reference, this is what I ran that worked to create and run with the overlay:
  - cd'd to the apptainer cache directory.
  - Created the image to overlay: apptainer overlay create --size 2024 comp24stimgate.img --fakeroot
  - --fakeroot necessary for ability to do things as root within the container, e.g. install packages
  - Ran with image as overlay: apptainer exec --fakeroot --overlay comp24st imgate.img comp24stimgate.sif bash
  - Neede to use exec rather than run as the rstudio base image wants to run rstudio in a server, and so you don't have access to the terminal

MiguelRodo commented 10 hours ago

Understanding the problem further

In GitHub codespaces, this is the path to the project library (i.e. this directory exists):

/workspaces/.local/lib/R/library/stimgate-9684ce2c

and renv does want to use it:

.libPaths()
[1] "/workspaces/.local/lib/R/library/stimgate-9684ce2c/linux-ubuntu-noble/R-4.4/x86_64-pc-linux-gnu"
[2] "/root/.cache/R/renv/sandbox/linux-ubuntu-noble/R-4.4/x86_64-pc-linux-gnu/25ebdc09"

On the HPC, however, this path does not exist, but a different one:

"/workspaces/.local/lib/R/library/stimgate-acc61424"

I am pretty sure the images used are the same. I've rebuilt the GitHub Codespace using a prebuild, and it's the same as before, and the image on the hpc was built this morning but the image in GHCR was last updated two days ago.

Okay, anyway, so that's the project library. We can install into it, no problem.

Allowing installation

So, to allow installation, we needed to change the RENV paths as follows in ~/.Renviron:

R_LIBS=/scratch/01463748/.local/lib/R
RENV_PATHS_LIBRARY_ROOT=/scratch/01463748/.local/lib/R/library
RENV_PATHS_CACHE=/renv/cache
RENV_PATHS_ROOT=/scratch/01463748/.local/lib/R/librar

Checking permissions in cache

The cache permissions are all full.

MiguelRodo / DevContainerFeatures