[guideline] design strategy

[bernard357]

Over past months a lot of code has been added to the initial structure of plumbery. There is now a need to re-think about the overall architecture. There are multiple reasons for this, and objectives pursued:

• facilitate the in-boarding of new developers
• embrace the coverage of multiple cloud service providers
• allow hybrid deployment of virtual servers and containers
• augment dramatically the coverage of tests
• make it easy to add a new action or to handle a small contribution
• breakdown the overall architecture in simple building blocks with simple interfaces (see below)
• introduce multiple sub-modules to arrange the code in small files neatly arranged

Feedback received from other developers is that the structure of plumbery is rather convulated. This is fully recognised. To improve in this area, here are basic design ideas that should drive next code refactoring efforts:

Data model -- each of the following should be implemented in separate classes:

• engine -- the running engine of plubery itself -- unique instance
• action -- the vocabulary understood by plumbery
• facility -- one per data centre and per cloud service provider
• blueprint -- named set of fittings, eventually spread across multiple facilities
• fitting -- service abstraction, e.g., 'domain', 'ethernet' or 'node'
• directive -- set of YAML settings, e.g. 'cloud-init' or 'powershell' or 'terradata'

Engine:

• only one running instance, defining a context of execution
• linked to a unique set of input data: manifest file / fittings plan / parameters file
• load and check all input files before running actions
• maintain a list of facilities
• accept actions and delegate execution to facilities
• maintain information shared across facilities (e.g., parameters)

Facility:

• one YAML document per facility
• each facility is bound to a single cloud service provider
• each facility is bound to one API endpoint (e.g., dd-eu, or dd-au, etc)
• can have settings specific to a provider (e.g., DD or AWS) or to back-end (e.g., PCEE vs public MCP)
• maintain a list of blueprints
• delegate the execution of actions to blueprints
• implemented in PlumberyFacility and PlumberyFalicityLoader -- to be created
• most of existing facility.py should be moved to facilities/dimensiondata.py
• new cloud service providers would be implemented there, e.g., facilities/aws.py -- like libcloud is doing

Blueprint:

• used as targets for action execution
• maintain a list of fittings that can be specific to a given cloud service provider
• has minimum scheduling capability to spread action execution on fittings
• implemented in new class blueprint.py -- to be created

Fitting:

• represent some abstraction of a service (e.g., 'domain', 'ethernet', 'node')
• implemented in PlumberyFitting and PlumberyFittingLoader -- to be created
• most of code in infrastructure.py and nodes.py should be moved to directory fittings -- to be done
• fittings/domain.py to implement network domains
• fittings/ethernet.py to implement VLAN
• fittings/node.py to implement node
• ... new abstractions such as containers, or meta-abstractions such as docker hosts
• pass the execution of actions to the directives

Directive:

• represent a set of configuration statements for fittings, e.g., 'glue', 'disks' ...
• implemented in PlumberyDirective and PlumberyDirectiveLoader -- to be created
• each class put in the directory directives
• directives/glue.py
• directives/disks.py
• directives/cloud-init.py
• directives/powershell.py
• directives/terradata.py
• ...
• all directives are initialised by the engine before running actions -- allow strong syntax checking
• handle a subset of actions passed to them -- invocation based on text labels directive.do('start')

Ok, that's enough information exposed at once. What do you think?

[bernard357]

Early implementation of actions and fittings in 1.0.1. This is not integrated with regular plumbery processing yet.

[bernard357]

Next step is to implement a blueprint class