.. image:: https://travis-ci.org/AustralianSynchrotron/Lightflow.svg?branch=master :target: https://travis-ci.org/AustralianSynchrotron/Lightflow
.. image:: https://readthedocs.org/projects/lightflow/badge/?version=latest :target: http://lightflow.readthedocs.io/en/latest :alt: Documentation Status
Lightflow is a Python 3.5+ library and command-line tool for executing workflows, composed of individual tasks, in a distributed fashion. It is based on Celery and provides task dependencies, data exchange between tasks and an intuitive description of workflows.
Python ^^^^^^ Lightflow is written in Python 3 and requires Python 3.5 or higher.
Operating system ^^^^^^^^^^^^^^^^ Lightflow is being developed and tested on Linux, with Debian and RedHat being the main platforms.
redis ^^^^^ The redis database is required by Lightflow as a communication broker between tasks. It is also used as the default broker for the Celery queuing system, but could be replaced with any other supported Celery broker.
MongoDB ^^^^^^^ Lightflow makes use of MongoDB for storing persistent data during a workflow run that can be accessed by all tasks.
The following getting started guide assumes a redis database running on localhost
and port 6379
as well as a MongoDB database running on localhost
and port 27017
.
Install Lightflow from PyPi::
pip install lightflow
Create a default configuration file and copy the provided example workflows to a local directory of your choice::
lightflow config default .
lightflow config examples .
If you like, list all available example workflows::
lightflow workflow list
In order to execute a workflow, start a worker that consumes jobs from the workflow, dag and task queues.
Then start a workflow from the list of available examples. The following example starts the workflow simple
::
lightflow worker start
lightflow workflow start simple