Workflow Systems, Dataflow Systems and Build Systems

[CMCDragonkai]

These 3 systems implies a more "batch" like workload rather than just running and maintaining networks of distributed services.

We would like to extend Matrix Emergence to handle these kinds of workloads that have different tradeoffs required.

• Workflow Systems - Are kind of like Business Process Management. These usually involve some level of human intervention stages. Examples of this include things that involve Trello.
• Dataflow systems - These are like preprocessing architectures that build data for ML systems. See Apache Airflow, Funflow, Luigi. Latency is not the biggest issue here, in most cases, high throughput is the most important aspect. But also likely to be handling very large state.
• Build systems - These are things that Make, Bazel... etc try to do.

This issue will track the research and implementation that goes into this:

• https://www.staff.ncl.ac.uk/andrey.mokhov/build-systems.pdf
• https://blogs.ncl.ac.uk/andreymokhov/build-systems-a-la-carte/
• https://www.tweag.io/posts/2018-07-10-funflow-make.html

[CMCDragonkai]

An example of batch task is an ML training system. Something that starts and ends, but doesn't run as a service. Any batch task in Matrix should be convertible to an event-driven service. But doing so requires consideration on the nature of the "event" (socket activated or otherwise), and how the batch Automaton starts and ends. One main issue is that the thing that starts the Automaton, does it represent a "recursive" representation of the Emergence orchestrator, that is, itself is an orchestrator, or is it relying on the global Emergence system to orchestrate the batch Automaton? Basically on a first-order basis, we think that the management of the lifecycle of Automatons is the job of the Emergence. But this only works if everything has the same global lifetime as each other. Batch tasks introduce the idea that an Automaton's lifetime is actually determined by application-level parameters.

Let's consider the tradeoffs of the 2 approaches:

• Recursive Emergence (similar to first class modules, we would imagine that the orchestrator itself is a first class feature, that is you can have Automatons that are themselves orchestrators)
• Global Emergence relying on binding/hardcoding/exposing the Emergence API to an Automaton that basically tells the Emergence system to bring up another Automaton. This however makes the relationship more implicit, and we no longer have a single source of control. A single control plane understandable and manipulated by the human operators interacting with the Architect language.

[CMCDragonkai]

There is an existing concept called "super server" https://en.wikipedia.org/wiki/Super-server that was the progenitor of socket activation in systemd http://0pointer.de/blog/projects/socket-activated-containers.html and http://0pointer.de/blog/projects/socket-activation.html.

It shows how event driven services can encapsulate other event driven services, and thus event driven services can definitely encapsulate batch tasks.

Note that xinetd and inetd was one of the first systems to implement on Linux.