Closed averagehat closed 8 years ago
Luigi might make it easier to handle making overall pipeline changes like
Also will allow automatic checkpointing, diagram construction, and allows stages to be run in chunks/independently.
Edit: An initial implementation might just be to wrap each script call in runsample.py in a luigi task. Related: http://ratatosk.readthedocs.org/en/latest/dev/introduction.html
runsample.py
Closing in favor of #155, as Luigi requires a running Tornado server and is more verbose. If we hit the limits of Make we can consider it among some other options.
Luigi might make it easier to handle making overall pipeline changes like
110 or #150
Also will allow automatic checkpointing, diagram construction, and allows stages to be run in chunks/independently.
Edit: An initial implementation might just be to wrap each script call in
runsample.py
in a luigi task. Related: http://ratatosk.readthedocs.org/en/latest/dev/introduction.html