This takes a DAG of "CallSpecs", and executes in a given namespace. There
are two methods of executing: either sequentially or in parallel. The
sequential implementation is very straightforward and is only 4 lines of
code. The parallel one is more complex. It uses the new Python 3.5 syntax
(that we target Python >= 3.5 is already stated in the doc). By default
each task is executed in a pool of processes and a subsequent task is
scheduled to run as soon as its dependencies are met.
Maybe we could use a decorator to specify whether a task is going to run in
another process or in the main one (blocking it but avoiding the cost of
serializing inputs and outputs and sending them by IPC).
The same implementation could be used to execute the tasks in different
nodes of a cluster, by supplying the appropriate executor.
This takes a DAG of "
CallSpecs", and executes in a given namespace. There are two methods of executing: either sequentially or in parallel. The sequential implementation is very straightforward and is only 4 lines of code. The parallel one is more complex. It uses the new Python 3.5 syntax (that we target Python >= 3.5 is already stated in the doc). By default each task is executed in a pool of processes and a subsequent task is scheduled to run as soon as its dependencies are met.Maybe we could use a decorator to specify whether a task is going to run in another process or in the main one (blocking it but avoiding the cost of serializing inputs and outputs and sending them by IPC).
The same implementation could be used to execute the tasks in different nodes of a cluster, by supplying the appropriate executor.