Allow more flexible creation of tasks

[colearendt]

Poking through the code a bit, it looks like the notion of "tasks" is how the workers actually get spawned / managed. If it is possible to either

(1) create a generic tasks R6 class that has the appropriate methods (farming those out to processx by default) for task management (2) allow processx to implement a more generic task backend

That would be great! Then I would expect users could provide their own tasks implementation, etc. Specifically, I am thinking about the following backends:

• background jobs in the RStudio IDE. This is a very nice way for even a single system to spawn and monitor the many tasks/jobs that may be generated here
• launcher jobs in RStudio Server Pro. If the RStudio Server Pro cluster is using a Kubernetes or Slurm cluster to launch R sessions, then it is possible that this feature would allow the revdepcheck access to many more resources than are available otherwise for parallelism. Further, we get some nice functionality within the IDE to manage this
• there are almost certainly other backends that people would like to use for HPC / distributed checking as well.

If this issue belongs upstream in processx, I'm happy to move it there!

[gaborcsardi]

This makes sense, but unfortunately no so simple to implement. The tasks make some assumptions about the package libraries being on the local machine, and the event loop can only handle processx subprocesses at this point. It is pretty hard to extend revdepcheck beyond the local machine.

[colearendt]

Yeah, that makes sense - thank you! The good thing is that the launcher jobs implementation actually uses an NFS share to keep the file system in sync. I am using launcher jobs for this behavior now, but I can only run a single container for the whole revdepcheck in the current setup (versus a container per worker / task).

LMK if it'd ever be helpful to have a walkthrough of what it looks like!

[hadley]

Unfortunately, I think this is out of scope for revdepcheck for the time being. If you (specifically) need to check many revdeps you can use #247)