arcalot / arcaflow-engine

Arcaflow is a highly-portable workflow engine enabling modular and validated pipelines through containerized plugins.
https://arcalot.io/arcaflow/
Apache License 2.0
6 stars 9 forks source link

Workflows with multiple outputs #72

Closed ghost closed 1 year ago

ghost commented 1 year ago

Please describe what you would like to see in this project

We need a way for workflows to have multiple possible outputs, much like steps. This would mean that the workflow definition could change like this:

outputs:
  success:
    some_data: !expr $.steps.some_step.outputs.success
  error:
    some_data: !expr $.steps.some_step.outputs.error

The output schema should be inferred from the content and the expression result type, but the workflow author may want to specify the output schema explicitly. For this purpose an output_schema key could be added. This schema would allow a workflow author to specify that a certain output should be considered an error and result in a non-zero exit code.

Please describe your use case

We will soon have #26 to iterate over a predetermined amount of data (e.g. servers in a cluster). In order to implement conditional loops (e.g. repeat until condition arises), we need to make workflows behave like steps and have multiple possible outputs. A conditional loop could then be implemented by specifying which output of a workflow should result in a repetition and which should result in a termination of a loop.

The implementation of this feature would also result in better error handling as workflow authors could pipe out error data while still maintaining a non-zero exit code and type safety.