Default processors for a workflow

[ajuvercr]

The existence of default processors is required to bootstrap real world workflow usage.

Pitch

A workflow following the connector architecture can provide arbitrarily complex behavior, but we have to start somewhere.

A common real-world workflow is something like this:

• Some agency provides a kafka channel with json objects denoting changes in some open data.
• A processor uses this data to created versioned json objects
• A RML processor created linked data member from these versioned objects
• A LDES Server Ingestor takes these members to be exposed as a LDES

This workflow needs the connector architecture because steps are in different programming languages.

Desired solution

• Defined kafka and ws channels.
• A JsRunner processor that executes a js function with instanced input and output streams.
• A RMLRunner that executes the RMLStreamer on an input stream and produces an output stream.
• A step config that runs with the JsRunner that creates versioned json objects
• A step that starts a RML process with a provided RML mapping file.
• A step that stores the members from an input stream into a LDES server.

Acceptance criteria

• The user can configure the source kafka stream to be the GRAR kafka stream.
• The user can configure the target LDES server.
• When the workflow runs, versioned LDES objects are stored in the LDES server.

Pointers

Examples of configuration (not linked data) and implementation can be found here https://github.com/ajvuercr/nautirust-configs

Scenarios

Manage workflows to derive/provide data to solid pods https://github.com/SolidLabResearch/Challenges/issues/50

[RubenVerborgh]

Can we apply this to a concrete use case, and have a specific demo, such that the challenge has a clear end?

[pheyvaer]

@ajuvercr Did you already have a chance to look into making the requested changes?

[ajuvercr]

This is the same suggested demo as #50 because that pipeline requires these default processors.

[pheyvaer]

@ajuvercr The acceptance criteria have to be more concrete. See here for an example.