search

MikeDacre / fyrd

Submit functions and shell scripts to torque and slurm clusters or local machines using python.
https://fyrd.science
MIT License
21 stars 8 forks source link

Merge python-pipeline into this project #9

Open MikeDacre opened 8 years ago

MikeDacre commented 8 years ago

The python-pipeline project is an effort to make it easy to create complex pipelines with python. It isn't that useful outside of a multithreading environment, so it makes sense to merge it in here and implement native multithreading in that project through the cluster module.

Rather than keep it as a separate project, the pipeline package should be added as a separate package alongside cluster to be used if the user wishes. However, it is important that its usage is not required in order to use the cluster package. ie. pipeline should depend on cluster, but cluster should not depend on pipeline.

MikeDacre commented 7 years ago

Most of the pipeline code is not needed, but add a Pipeline class that jobs can be added to.

This class should write itself to disk and be easily restorable from the name and directory alone. All jobs will have the track option in #80 set, which will allow pipeline restoration.