crew: a distributed worker launcher framework

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The NNG-powered mirai R package is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The crew package extends mirai with a unifying interface for third-party worker launchers. Inspiration also comes from packages future, rrq, clustermq, and batchtools.

📣 Request for community contributions 📣

The launcher plugin framework aims to extend crew to modern platforms and services like Google Cloud Run, Kubernetes, and beyond. At the time of writing, plugins for cloud computing do not yet exist. So if you have access to these services and know how to use them, please consider contributing a package with plugins of your own. The tutorial at https://wlandau.github.io/crew/articles/plugins.html explains how. The maintainer of crew would love to work with you!

Installation

Documentation

The documentation website at https://wlandau.github.io/crew/ includes a function reference and tutorial vignettes linked below.

1. Introduction to crew
2. Controller groups
3. Integration with Shiny
4. How to write launcher plugins
5. Known risks of crew

Risks

The crew package has unavoidable risks, and the user is responsible for safety, security, and computational resources. Please read the software license and the vignette about specific known risks.

Similar work

• mirai: a powerful R framework for asynchronous tasks built on NNG. The purpose of crew is to extend mirai to different computing platforms for distributed workers.
• rrq: a task queue for R based on Redis.
• rrqueue: predecessor of rrq.
• clustermq: sends R function calls as jobs to computing clusters.
• future: a unified interface for asynchronous evaluation of single tasks and map-reduce calls on a wide variety of backend technologies.
• batchtools: tools for computation on batch systems.
• targets: a Make-like pipeline tool for R.
• later: delayed evaluation of synchronous tasks.
• promises: minimally-invasive asynchronous programming for a small number of tasks within Shiny apps.
• callr: initiates R process from other R processes.
• High-performance computing CRAN task view.

Thanks

The crew package incorporates insightful ideas from the following people.

• Charlie Gao created mirai and nanonext and graciously accommodated the complicated and demanding feature requests that made crew possible.
• Rich FitzJohn and Robert Ashton developed rrq.
• Gábor Csárdi developed callr and wrote an edifying blog post on implementing task queues.
• Kirill Müller created the workers prototype, an initial effort that led directly to the current implementation of crew. crew would not exist without Kirill’s insights about orchestration models for R processes.
• Henrik Bengtsson. Henrik’s future package ecosystem demonstrates the incredible power of a consistent R interface on top of a varying collection of high-performance computing technologies.
• Michael Schubert. Michael’s clustermq package supports efficient high-performance computing on traditional clusters, and it demonstrates the value of a central R6 object to manage an entire collection of persistent workers.
• David Kretch. The paws R package is a powerful interface to Amazon Web Services, and the documentation clearly communicates the capabilities and limitations of AWS to R users.
• Adam Banker, co-authored paws with David Kretch.
• David Neuzerling. David’s lambdr package establishes a helpful pattern to submit and collect AWS Lambda jobs from R.
• Mark Edmondson. Mark maintains several R packages to interface with Google Cloud Platform such as googleCloudStorageR and googleCloudRunner, and he started the conversation around helping targets submit jobs to Google Cloud Run.
• Joe Cheng for sparking the integration of crew with promises.

Code of Conduct

Please note that the crew project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Citation

To cite package ‘crew’ in publications use:

  Landau WM (2023). _crew: A Distributed Worker Launcher Framework_.
  https://wlandau.github.io/crew/, https://github.com/wlandau/crew.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {crew: A Distributed Worker Launcher Framework},
    author = {William Michael Landau},
    year = {2023},
    note = {https://wlandau.github.io/crew/, https://github.com/wlandau/crew},
  }