Proposal: modular, dataflow-based architecture for the LeApp CLI

[pcahyna]

Synopsis

It is proposed to create a more modular architecture for the LeApp CLI, using dataflow-based programming concepts and frameworks. Modules in this architecture are called Actors. This infrastructure is also intended to serve as the next generation of the preupgrade-assistant tool, whose current modular architecture is too restricted. Actors will specify what types of data they require and provide on their input and output ports respectively, and a dependency solver will produce a dependency graph to satisfy those requirements, i.e. connect output ports to appropriate input ports. We considered two frameworks for flow-based programming: WOW:-P and Selinon. The former has the advantage of being simpler and apparently better expressing the dependency information that we need. The advantage of Selinon on the other hand is that it is designed to cope with non-idempotent actors. The actual implementation of actors will need to be possible to be done in shell, with a Python wrapper to present the proper interface for the (Python-based) framework. It is furthermore proposed to execute those shell parts using Ansible (as Ansible modules) and let them communicate using JSON, as it is the standard for Ansible modules.

Goals

• create a modular architecture for LeApp, so that the information about various migration strategies and application-specific logic is not kept in a single source file but in independent "modules" or "plugins", which can eventually be maintained by different people, even by experts on a particular application who are not part of the core LeApp team.
• share this infrastructure with preupgrade-assistant, to serve as a basis of a new generation of preupgrade-assistant. Current preupgrade-assistant has a modular architecture and lots of modules have been already written, but the infrastructure is limited. In particular, there are no dependencies between modules and no way to pass information between modules, so if several modules need the same information, the code which produces it can not be a module itself, as it has no way to pass this information to them.

  Terminology

  As "module" is a very overloaded word, we will use the term "actors" for the components in the proposed framework. (This is a term borrowed from one of the infrastructures considered - WOW:-P - in similar infrastructures this concept may be also referred to as "tasks".) It should be still kept in mind that they correspond to modules in the current version of preupgrade-assistant. (We will need to reserve the word "module" for Ansible modules later.) Actors communicate using interfaces called "ports", they accept inputs on input ports and provide outputs on output ports. The list of input and output port are a part of the definition of an actor.

  Requirements

• Dependencies between actors must be possible to be expressed. (I.e. one must be able to express that one actor should be run before the other.)
• Actors must be able to pass data to other actors.
• Dependencies between the actors should be expressed as data dependencies. I.e. an actor depends on another actor if and only if it needs some data from the latter. This makes possible to determine what actors to run by walking backwards from the desired actions (i.e. I want to run X, but X needs a piece of data xy from Y, so I need to run Y first and pass xy from it to X). If nothing needs the output of an actor, it does not need to be run. This requirement integrates the previous two in a way that prevents duplication of information - one will not have to specify separately that X depends on Y and that X needs xy from Y, this will be done in one step.
• Data dependencies should be specified implicitely. In the target situation, there will be lots of actors. (The current preupgrade-assistant has about 132 modules already!) In this situation specifying all the dependencies explicitely ("X needs xy from Y") does not scale. We need to be able to specify for each module what it neeeds ("X needs xy") and what it provides ("Y provides xy") and the infrastructure shall made the connection from Y to X ("X needs xy from Y") automatically based on this information.
• It should be possible to implement actors as shell scripts. It is foreseen that actors would be written by specialists for the given application/area that the actor will handle, but such specialists are not necessarily Python programmers, usually they can write a shell script though. In the current preupgrade-assistant, actors can be implemented either in Python or in shell.
• Would be nice to have command execution done by Ansible, because that's an infrastructure in place already for this type of task and we do not want to reinvent the wheel.

  Proposed solution

  The proposal is to implement LeApp using a flow-based programming infrastructure. We considered several flow-based infrastructures:

• Selinon/Celery http://selinon.readthedocs.io/en/latest/
• WOW:-P https://bitbucket.org/urbanj/wowp
• pyutilib.workflow https://github.com/PyUtilib/pyutilib/

For the initial implementation we chose WOW:-P. Its advantages over Selinon are:

• Selinon is Python 3 - only and it is not completely clear whether Python 3 will be available on the systems which are the target of this project.
• WOW:-P is more lightweight - data are passed in-process - actually everything can be in a single process, making it ideal for rapid prototyping and for implementing a self-contained tool such as the current LeApp CLI. Selinon needs a database engine to store data shared between actors (tasks): http://selinon.readthedocs.io/en/latest/storage.html. (However, one can implement a custom storage adapter, and it should be easy to implement one as a Python class fully contained in the process, if one does not need distributed execution. The underlying Celery library also needs a database adapter http://selinon.readthedocs.io/en/latest/faq.html#do-i-need-result-backend but Selinon has a "simulator": http://selinon.readthedocs.io/en/latest/faq.html#can-i-simulate-selinon-run-without-deploying-huge-infrastructure which apparently can run the flow from CLI without using Celery. This could be the way to use Selinon purely in-process and be similar in simplicity to WOW:-P.)
• In Selinon, dependencies between tasks are separate from the data flow, i.e. the edges between nodes (tasks, actors) of the dependency graph do not correspond to the flow of data. represent As we prefer dependencies to be data dependencies, this would have to be implemented as a layer above Selinon.

It may turn out that in the future we will need Selinon features in which case it may be preferable to switch to it. In particular, we may need it if we want to allow non-idempotent actors. In a simple implementation of the workflow, if its execution is aborted due to a failure of an actor, the whole process must be restarted. This is not acceptable in the case when some of the actors are non-idempotent. In Selinon, tasks which have been completed in a previous run can be skipped: http://selinon.readthedocs.io/en/latest/selective.html#reuse-results-in-selective-flows It is thus a matter of decision whether we will forbid non-idempotent actors to simplify the framework, or allow them in order to potentially simplify some of the actors. In the latter case Selinon will be probably the easiest solution and a layer which produces Selinon dependencies from data dependencies will need to be implemented. pyutilib.workflow seems to be very similar to WOW:-P and choosing between them is a matter of taste. So far we choose WOW:-P because @pcahyna is in contact with its developers from previous job. None of the infrastructures evaluated has implicit data dependencies. In WOW:-P and pyutilib.workflow one has to connect explicitely the source port of one actor to target port in another actor to pass data between them and establish the dependency. It is therefore needed to write a dependency resolution engine which converts implicit dependencies to explicit connections between actors. The usefulness of the infrastructure will in a large part depend on how well will it be possible to express those dependencies. In the initial design, the output ports of actors have an "annotation" property which contains the data type that they provide. The input ports also have an "annotation" property which contains the data type that they require and optionally the name of the source actor that they require it from. In the dependency resolution step, the resolver connects input ports to output ports whose provided data type is a subtype of their required data type and whose actor's name correspond to their required one (if any). At the end of the resolution phase, if an input port is connected to more than one output port, it is considered an error. As an exception, there may be input ports marked as accepting multiple inputs. In this case the framework will merge all the inputs into a list and this port will obtain this list. This is important for preupgrade-assistant, which ultimately produces a report from all the actors. Therefore, each actor shall have an output port which provides its status to the report-generating actor. The report generating actor shall have an input port which accepts the "status" data type from any actor and accepts multiple inputs. It will then obtain the statuses from all the actors and produce a report from them. The actors in WOW:-P (and other Python-based workflow infrastructures) are just Python objects. As we want to be able to write actors as shell scripts, we will need a Python class which will wrap such a script in a proper Python actor. We will then need a way to get output data (for output ports) from the shell script, which is a bit difficult, as shell script have poor possibilities of providing outputs - only stdout, stderr and possibly temporary files. In Ansible, modules can provide a set of structured data, called facts. Moreover, an Ansible module is just a script written in any language, including shell, which Ansible executes on the target system. It therefore has all the properties which we need from our script which implements an actor. Therefore, our actors should look like a Python wrapper class (to provide the proper interface required from an actor in the chosen workflow infrastructure such as WOW:-P) which internally calls Ansible, which executes a module, which can be a shell script. The wrapper class in Python should be the same for most, if not all, actors. Those who will implement the actors will thus only need to provide the Ansible module (shell script). The output (facts) from an Ansible module is done using JSON. We thus shall restrict the data passed between actors to what can be easily represented by JSON, although at least WOWP:-P can pass any Python objects. There will probably need to be helper shell functions for the JSON output formatting and other tasks common to all the actors' scripts.

[ncoghlan]

This general approach sounds reasonable to me, so I think it makes sense to start moving in this direction.