Rethink Bob core

[lispyclouds]

Check out Vert.x as a transport When clj-docker-cleint is async, go for it Check out async adapters for HugSQL

[lispyclouds]

First draft of the new proposed architechture:

[lispyclouds]

Considerations:

• Use of RabbitMQ as the durable queue
• Cluster to perform CRUD ops for pipelines, artifact stores and resource providers
• Cluster of pipeline runners
• Cluster of loggers
• Health check cron
• Each of the components should have a maintenence endpoint?
  • This would mean each one of them also needs to be aware of maintainence messages which would need some self introspection

[bitsapien]

What happens when a pipeline "start" follows a pipeline "create", and your CRUD Entity systems is overloaded and never persisted the "pipeline definition" ?

[lispyclouds]

In that case the general flow is:

• Since all the services are independent, each one of them tries to work with what is available.
• If the start of a pipeline is requested before it exists, the runner picking it up will simply error out and put a message on the error queue.
• This error queue is to be exposed via a REST path and its upto the users discretion to decide what to do with each of the errors. Errors are quite subjective too i think.

[lispyclouds]

Being developed on this branch

[lispyclouds]

The logging service is not to be used for simplicity reasons.

[lispyclouds]

Since its a distributed service architechture, consider using a schemaless DB as a value store and not postgres to avoid shared DB state issues

[lispyclouds]