Bioconductor Single Package Builder (SPB)

Overview

The single package builder runs R CMD build and R CMD check on packages and provides the output of these processes. The system consists of the EC2 instances staging.bioconductor.org, issues.bioconductor.org and rabbitmq as well as the current devel build machines. The flow of information starts at issues.bioconductor.org triggered by a new package submission or update (version bump). A message is sent to rabbitmq which sends messages to the build nodes. staging.bioconductor.org tracks build progress and records logs.

Code

Code for the SPB is stored in GitHub.

Currently, the SPB consists of 2 components in this git repo, in top-level directories:

• spb_history - A Django web app to track build history. Accessible at http://staging.bioconductor.org:8000
• workers - Python scripts that run on each build machine

Options to start a build

1. Submitting packages to github
2. Manually restart a build.

These all send messages to an installation of RabbitMQ (a Java-based messaging framework) to listeners on each build machine (server.py). The build machines start building the package and send back progress messages.

There is another listener (called archiver.py which writes build events to a database, where they can then be displayed by the spb_history web application (written in Python's Django framework).

Where possible, the code uses existing code from the Bioconductor Build System (BBS). In particular, it uses BBS scripts to set environment variables for the build, though these are overridden in a couple of cases.

On the build machines, the listener that is always running is called server.py. When it receives a build request, it kicks off a script particular to that build machine (called e.g. petty.sh or moscato2.bat) which sets environment variables, then runs builder.py to do the actual building.

Build Machines

See active_devel_builders in http://bioconductor.org/config.yaml to see which machines are used by the SPB.

System Integration

A job is submitted to the Single Package Builder (SPB) when a package is submitted to the github contributions (https://github.com/Bioconductor/Contributions) or when the version of an existing package is bumped.

• New Package Submission

The code that manages when issues (packages) are submitted can be found at issue_tracker_github and is implemented on the AWS cloud instance issues.bioconductor.org.

• Version bump

When a package submits a version bump, the above manager is run. It sends a message to the listeners on each build node as described at the start of this document.

In both cases, the message sent to the SPB contains a flag indicating that this build was originated by the SPB.

When the build is complete, another script on staging track_build_completion.py is listening, and it posts a message to the contribuctions issue (using an HTTPS request) including a link to the build report.

• SPB Moving Parts

The SPB was designed to use a message broker (in this case RabbitMQ) so that components could be loosely coupled and so that they could send information in an asynchronous, real-time fashion, allowing live updates.

In practice there are some dependencies between the moving parts.

Here are the moving parts, first briefly and then in more detail.

• Broker - currently a machine in the cloud called rabbitmq. All it does is run an RabbitMQ message broker. Messages are passed using a text protocol called STOMP.
• Issue manager. Currently implemented on the machine issues.bioconductor.org. It detects when a package is submitted, or updated, and sends a message to the SPB telling it to start a build.
• build node server; a python script called server.py should be running on each build node at all times. When an instruction is received to start a build, this script starts a new process (first sourcing some variables specific to the build node, then running a python script called builder.py).
• Three scripts that run on the staging.bioconductor.org machine:
  1. track_build_completion.py: monitors progress of builds and when a build is complete (all machines have finished) it posts the build report to the web and posts a message to the issue.
  2. archiver.py: monitors progress of builds, writes all build info to a database to be displayed by a Django web app (what you see when you hit http://staging.bioconductor.org:8000/).
  3. rerun_build.py: for manually kicking off an SPB build without having to version bump or resubmit.