prefactor-cwl

CWL version of prefactor

https://github.com/lofar-astron/prefactor/

Note that this project is still very much in alpha stage and very much in flux. It is a prototype implementation and is not intended for end user audience. The goal is to demonstrate various technologies and how they could assist simplifying construction and deployment of big scale data reduction pipelines.

requirements

• Python Virtualenv

optional:

• Slum
• Docker
• Singularity

usage

To run the pipeline with the example dataset just run:

$ make run

To Run using a udocker (user space docker) run:

$ make run-udocker

To try out toil as a schedular run:

$ make toil

HPC cloud

We've made a set of ansbile scripts to automate the deployment of a cluster that can run CWL pipelines: https://github.com/EOSC-LOFAR/ansible

on Cartesius

If you are on cartesius, you first need to activate a python environment:

$ module load python

Since all jobs are quite small, it is better to use the staging partition. You can configure toil to use the staging partition with:

$ export TOIL_SLURM_ARGS="-t 0:30:00 -p staging"

$ make slurm

steps

the pipeline consists of these steps: