search

workflux / workflUX

An open-source, cloud-ready web application for simplified deployment of big data workflows.
Apache License 2.0
33 stars 8 forks source link
bigdata bioinformatics common-workflow-language web-application workflow workflows

readme

workflUX - The Workflow User eXperience

(Formerly known as CWLab.)

An open-source, cloud-ready web application for simplified deployment of big data workflows.

CI/CD: Build Status

Packaging: PyPI status PyPI version shields.io PyPI pyversions Docker Cloud Automated build

Citation & Contribution: DOI All Contributors

Installation and Quick Start:

Attention: workflUX is in beta state and breaking changes might be introduced in the future. However, if you like to test it or even run in production, we will support you.

Installation can be done using pip:
python3 -m pip install workflux

Please see the section "Configuration" for a discussion of available options.

Start the webserver with your custom configuration (or leave out the --config flag to use the default one):
workflux up --config config.yaml

If you like to make use of containers for dependency management, you need to install Docker or a Docker-compatible containerization solution like singularity or udocker. To run on Windows or MacOs, please install the dedicated docker versions: Docker for Windows, Docker for Mac

The usage of the web interface should be self-explanatory with build-in instruction. The following section gives an overview of the basic usage scenario.

Supported Systems:

workflUX is written in platform-agnostic python and can therefore be executed on:

Any CWL runner that has a command-line interface can be integrated into workflUX in order to execute CWL workflows or tool-wrappers, such as:

Therefore, workflUX can be used on any infrastructure supported by these CWL runners, including:

*Please Note:
Execution on Windows is only supported by cwltool which talks to docker for windows. Therefore, CWL-wrapped tools and workflows which where originally designed for Linux/MacOs can be executed on Windows with a graphical interface provided by workflUX.

Usage:

Please see our tutorial, that walks you through an simple yet meaningful example of how workflUX can be used to compare the spike protein sequences of Covid-19 in two patient cohorts.

Here are some apetizers: welcome screenshot

create job screenshot

Configuration:

workflUX is a highly versatile package and makes almost no assumptions on your hard- and software environment used for the execution of CWL. To adapt it to your system and use case, a set of configuration options is available:

All configuration options can be specified in a single YAML file which is provided to workflUX upon start:
workflux up --config my_config.yaml

To get an example config file, run the following command:
workflux print_config > config.yaml (or see the example below)

General Configs:

Exec Profiles:

This is where you configure how to execute cwl jobs on your system. A profile consists of four steps: prepare, exec, eval, and finalize (only exec required, the rest is optional). For each step, you can specify commands that are executed in bash or cmd terminal.

You can define multiple execution profile as shown in the config example below. This allows frontend users to choose between different execution options (e.g. using different CWL runners, different dependency management systems, or even choose a between multiple available batch execution infrastructures like lsf, pbs, ...). For each execution profile, following configuration parameters are available (but only type and exec is required):

* Additional notes regarding execution profile steps:

Example configuration files:

Below, you can find example configurations for local execution of CWL workflows or tools with cwltool.

Linux / MacOs:

WEB_SERVER_HOST: localhost 
WEB_SERVER_PORT: 5000

DEBUG: False  

TEMP_DIR: '/home/workflux_user/workflux/temp'
WORKFLOW_DIR: '/home/workflux_user/workflux/workflows'
EXEC_DIR: '/datasets/processing_out/'
DEFAULT_INPUT_DIR: '/home/workflux_user/workflux/input'
DB_DIR: '/home/workflux_user/workflux/db'

ADD_INPUT_DIRS:
    GENOMES_DIR: '/ngs_share/genomes'
    PUBLIC_GEO_DATA: '/datasets/public/geo'

ADD_INPUT_AND_UPLOAD_DIRS:
    UPLOAD_SCRATCH: '/scratch/upload'
    PERMANEN_UPLOAD_STORE: '/datasets/upload'

EXEC_PROFILES:
    cwltool_local:
        type: bash
        max_retries: 2
        timeout:
            prepare: 120
            exec: 86400
            eval: 120
            finalize: 120
        exec: |
            cwltool --outdir "${OUTPUT_DIR}" "${WORKFLOW}" "${RUN_INPUT}" \
                >> "${LOG_FILE}" 2>&1
        eval: | 
            LAST_LINE=$(tail -n 1 ${LOG_FILE})
            if [[ "${LAST_LINE}" == *"Final process status is success"* ]]
            then
                SUCCESS=True
            else
                SUCCESS=False
                ERR_MESSAGE="cwltool failed - ${LAST_LINE}"
            fi

Windows:

WEB_SERVER_HOST: localhost
WEB_SERVER_PORT: 5000

DEBUG: False  

TEMP_DIR: 'C:\Users\workflux_user\workflux\temp'
WORKFLOW_DIR: 'C:\Users\workflux_user\workflux\workflows'
EXEC_DIR: 'D:\processing_out\'
DEFAULT_INPUT_DIR: 'C:\Users\workflux_user\workflux\input'
DB_DIR: 'C:\Users\workflux_user\workflux\db'

ADD_INPUT_DIRS:
    GENOMES_DIR: 'E:\genomes'
    PUBLIC_GEO_DATA: 'D:\public\geo'

ADD_INPUT_AND_UPLOAD_DIRS:
    UPLOAD_SCRATCH: 'E:\upload'
    PERMANEN_UPLOAD_STORE: '\D:\upload'

EXEC_PROFILES:
    cwltool_windows:
        type: powershell
        max_retries: 2
        timeout:
            prepare: 120
            exec: 86400
            eval: 120
            finalize: 120
        exec: |
            . "${PYTHON_PATH}" -m cwltool --debug --default-container ubuntu:16.04 --outdir "${OUTPUT_DIR}" "${CWL}" "${RUN_INPUT}" > "${LOG_FILE}" 2>&1

        eval: |
            $LAST_LINES = (Get-Content -Tail 2 "${LOG_FILE}")

            if ($LAST_LINES.Contains("Final process status is success")){$SUCCESS="True"}
            else {$SUCCESS="False"; $ERR_MESSAGE = "cwltool failed - ${LAST_LINE}"}

Licence:

This package is free to use and modify under the Apache 2.0 Licence.

Contributors ✨

Thanks goes to these wonderful people (emoji key):


Kersten Breuer
💻 🎨
Pavlo Lutsik
💻 🤔 💵
Sven Twardziok
💻
Marius
💻 🚇
Lukas Jelonek
💻
Michael Franklin
💻
Alex Kanitz
💻

Yoann PAGEAUD
💻
Yassen Assenov
🤔
YuYu Lin
💻 🔌

This project follows the all-contributors specification. Contributions of any kind welcome!