A spec/ontology to describe the workflows used by the orchestrator

[ajuvercr]

Create onthology for components in a workflow.

Pitch

A workflow can only be as powerful as the combined power of its steps. To not .... the power of a workflow, it is paramount that steps can be implemented in any way, shape or form (different programming languages, one or many machines etc). To make this possible an architecture (coined connector architecture) was introduced, where each step corresponds to a processor, and processors communicate via channels.

Existing frameworks for workflow management, such as NiFi, Oozie, Airflow, and Dagster restricts the users within the context of the frameworks, be it in terms of programming language, limited API extensibility or fixed orchestration mechanism. On the other hand, DSL based workflow management tools such as Toil and Snakemake are limited in the tasks that they support which includes only BASH scripts.

Nextflow solved the aforementioned problems of the workflow management systems, however, it only supports file-based channels for data transfer. It cannot set up a workflow with processors using arbitrary channels such as Kafka for data transfer.

Optionally: A processor can be viewed as a runner. A runner abstracts the connector architecture away from the end user but requires some configuration. An example runner can be a JsRunner that takes a path to a source file and a function name and boots up the required channels and passes them to the expected function.

Desired solution

• A model to describe channels
  • This model can be shapelike, instances of the channel can be validated according to the channel. (for example define a File channel, this channel can be instanciated with a config that has a field path which makes the config valid)
  • Channels can have a startup (or prepare) procedure (for example start a docker container that creates a kafka instance)
  • Channels can have a shutdown (or cleanup) procedure (for example shut down this docker container)
• A model to describe the processors
  • Processors have to be able to indicate what channels are supported
  • This model can be shapelike, instances of the processor can be validated according to the processor. (for example define a JsRunner processor, this processor can be instanciated with a config that specifies a path and a functionName, which makes the config valid)
  • Processors can have a startup (or prepare) procedure (for example compile source code)
  • Processors can have a shutdown (or cleanup) procedure
• A model to describe a step
  • A step has to be able to indicate the expected input and output channels
  • A step has to be able to indicate what processor can execute the step

Acceptance criteria

• There exists a channel ontology that can describe the following channels
  • File
  • Kafka
  • Websockets
• There exists a processor ontology that can describe the following processors
  • JsRunner processor
  • RML processor
• There exists a step ontology that can describe the following steps
  • HelloWorld (step that requires no input or output streams)
  • RMLStep (step that requires a RML File path, a input stream and a output stream)

Pointers

Examples of configuration (not linked data) can be found here https://github.com/ajuvercr/nautirust-configs

Scenarios

Manage workflows to derive/provide data to solid pods https://github.com/SolidLabResearch/Challenges/issues/50

[RubenVerborgh]

Can we apply this to a concrete use case, and have a specific demo, such that the challenge has a clear end?

[pheyvaer]

@ajuvercr Did you already have a chance to look into making the requested changes?

[ajuvercr]

It is difficult to give a demo about an ontology. On the other hand challenges like #50 require this challenge, which makes this challenge not really a SolidLabs Challege?

[pheyvaer]

Yeah fair point. @RubenVerborgh What do you think?

[RubenVerborgh]

I think there should be one or a couple of example workflows that can be described by (version 1 of) this ontology; this will give us an idea of what the ontology requires.

So basically, I am looking for any reasonable/representative workflow that, if the ontology can describe it, we can call the ontology a success.

[pheyvaer]

@ajuvercr Can you implement the necessary changes?

[pheyvaer]

@ajuvercr Do you think you can add what Ruben suggested?