Add the ability to run Kale CommandLineTasks via Parsl. Bash and Python tasks can now be run seamlessly in the same workflow, and data can be passed between them (at least Bash -> Python. Python -> Bash probably doesn't work yet.)
I've chosen to use parsl's Python tasks and the subprocess module to execute kale CommandLineTasks in order to use parsl's AppFutures, which seem to be available only to Python tasks. Relying only on DataFutures from parsl bash tasks, it seems that dependencies can be communicated only through file existence, whereas I wanted to retain dependency based on process completion.
It is not yet possible to create a Kale CommandLineTask from a parsl bash task. Despite the previous paragraph, it should be straightforward to extract the data from a parsl bash task to create a kale CommandLineTask, but they will still be run as parsl python tasks upon execution, so data dependencies will be dropped.
Add the ability to run Kale CommandLineTasks via Parsl. Bash and Python tasks can now be run seamlessly in the same workflow, and data can be passed between them (at least Bash -> Python. Python -> Bash probably doesn't work yet.)
I've chosen to use parsl's Python tasks and the
subprocessmodule to execute kale CommandLineTasks in order to use parsl'sAppFutures, which seem to be available only to Python tasks. Relying only onDataFuturesfrom parsl bash tasks, it seems that dependencies can be communicated only through file existence, whereas I wanted to retain dependency based on process completion.It is not yet possible to create a Kale CommandLineTask from a parsl bash task. Despite the previous paragraph, it should be straightforward to extract the data from a parsl bash task to create a kale CommandLineTask, but they will still be run as parsl python tasks upon execution, so data dependencies will be dropped.
See
Parsl Bash.ipynbfor a demo.