Build a new API for Workflows

[sapetnioc]

We should define a backend agnostic Workflow structure that would be produced by Capsul given an executable, an engine and an execution context (i.e. a configuration of the engine for the given executable). This workflow structure would be given to an execution backend that would have to start jobs and wait for some jobs termination to start other jobs. A first implementation of such an execution backend could be easily done using Soma-Workflow. In this issue, we have also to define the communication API between Capsul and the execution backend.

A workflow would be composed of:

• Files to copy from the client environment to the execution environment. These files are not user data files, there are internal utility files used to send information to jobs, to communicate between jobs and to get jobs output. There destination is a temporary directory dedicated to the workflow execution that must be accessible to all execution nodes. Therefore, this is a strong requirement of the execution backend: being able to create a temporary directory accessible to every execution jobs.
• A directed acyclic graph of command lines to execute. Each edge in the graph meaning do not start the target command before the source command is finished.
• Environment variables to set globally for all command lines. I do not know yet if we will need to be able to customize environment variables per command line.
• Utility files to copy from the execution environment to the client environment once the workflow execution is finished (whether successful or not).

[denisri]

Is it really a need for a "new" API ? Or a matter of picking an existing one and using this one ? There are already a number of existing APIs around (as you have already looked at it), and soma-workflow has his one which suits our needs, even if no other did the job. So I'm not sure we need to reinvent the wheel once more ?

[sapetnioc]

This is the idea I presented in a meeting weeks ago. To be able to use soma-workflow or other existing engines, we need to decide what input to give to the workflow engine. Not all engines have the same features but some features of soma-workflow such as temporary files management are already managed by Capsul context. Therefore we can rely on a set of workflow features common to all engines. But we must define how we will create an engine specific workflow using the new API. In other words how to create a workflow (either Soma-workflow or other) given an executable, a Capsul engine and an execution context.