REP 2: Ralph modules architecture

[vi4m]

Introduction

According to the Zen of Ralph we should avoid introducing new requirements for the main ralph package.

Recently we started work on generating PDF reports / transition outprints which doesn't cover ralph scope itself.

Therefore i propose simple and flexible architecture for separating concerns, following industry standards microservices architecture to separater further ralph modules to separate processes.

Ralph extensions architecture

Every new ralph extension, such as inky connector should be separated from main ralph package.

1. Separate repository with the ralphext prefix
2. Separate process, accessible from any network address
3. Architecture doesn't depend on concrete microservices transport, such as nameko, lymph, or raw redis invocations
4. Shared configuration using etcd daemon

   Example - inkpy

5. We create github repository called ralph_ext_inkpy with custom dependencies. This package doesn't depend on ralph itself, we don't want circural dependencies.
6. We introduce extensions index in the documentation which explains what this extensions does.
7. To simplify adoption of microservices, let's start with something simple. I thereby propose using just raw redis - rq sync queues which gives us separation of processes, boundaries of services, but on the other hand - doesn't introduce any new complex dependency such as rabbitmq, or zookeeper to maintain, and should be sufficient for our needs.

[mkurek]

:+1: for using just raw redis.

We could use redis not only as job queue (we're using it right now through RQ - is simple, but covers all our requirements) - we could use it as a message broker/cache in microservices world.

Inkpy is IMO not perfect example for presenting microservices architecture - inpky could be used as (synchronous) job executor using just RQ - everything we need is to schedule job on specific queue (ex. ext_pdf) with template and data passed. Inkpy should be plugged into redis (using python-rq) and listed on this queue. It's simple usage of rq jobs just like in Ralph 2.

If we want to have real microservices (ex. display content from external service), we could also use redis to do this (assuming for example only static content displayed by external service).

Proposed requirements:

• content from external (micro)service is displayed on separated tab attached to model view (using existing extra views mechanism) - it's only static content
• every service which want to display content in Ralph should be approved by Ralph maintainers and authorized (using appropriate data in settings, ex. service auth token and possible models/tabs to work on)
• when user access particular model page, service should provide information if it's tab is accessible for this object (ex. there is security tab for DataCenterAsset, but external service has only information about servers security - tab is hidden (and not accessible) for example for switches etc.)
• when service tab is opened, service should provide template and data to render
• every time service sould have passed serialized object

There are few possibilities to achieve this:

1. request-response mechanism: when tab is opened, we could call external service directly to get template and data to display (using synchronous rq job to act like rpc); this is the simplest mechanism with severals drawbacks (ex. latency, dependency from external service performance) but when connected with cache (using redis or memcached) it could be enough for us
2. pub-sub mechanism (using redis pub-sub): ralph is subscriber here (for every registered service), external service is publisher and it's deciding whether something changed and should be published again. When tab is accessed, there is no direct call from Ralph to external service - everything is assumed to be stored in redis (we should use some cache again to store received messages).
3. raw cache mechanism: since pub-sub is overkill in our case (there will be only one subscriber - Ralph and we're only interested in the most recent version of each message (per object)) we could use just raw redis cache to store this data. Service is pushing data to redis cache, Ralph is reading data to display from cache - thats all!

Notice that option 1 and 3 are quiet similar, with advantage in option 1 when there is no persistance - when something is not in cache just call external service to get it.

To consider deeply:

• should service have readonly access to database (directly or by some RPC through Ralph)?
• should every object be exposed in redis to be accessable by external service (compare with DB access)? Or maybe REST API is enough?
• persistance - should cache/messages be stored persistently or not?
• security - should each service has it's own redis db (to not interfere with other service messages/cache)?
• how to easily allow new services to publish in Ralph (with minimal effort in redis, ex. changing configuration file), but staying secure at the same time?
• should data provided by external service be accessible through Ralph REST API?

@andrzej-jankowski @quamilek please rate this ideas too (maybe I'm missing something about redis etc).

[ar4s]

Done