B2SHARE microservices oriented deployment

[hjhsalo]

In order to improve deployment options (currently just Docker-Compose) we should define B2SHARE architecture; supportive services (Elasticsearch, Database / PostgreSQL, Message queue / RabbitMQ, Redis) B2SHARE workers (web, background) and routing / load balancing.

Let's model B2SHARE according to what Invenio currently recommends: (https://github.com/inveniosoftware/invenio-app-ils/blob/master/docker-compose.full.yml | https://github.com/inveniosoftware/invenio-app-ils/blob/master/docker-services.yml | https://github.com/inveniosoftware/invenio-app-ils/blob/master/docker-compose.yml)

• Generic naming of support services (db == PostgreSQL, frontend == NGINX, etc.)
• Define REST API service, background worker service, etc.
  • If possible define a single image and specialize instance by using different startup parameters (e.g. uwsgi for REST API, celery for background worker)

Update current B2SHARE docker-compose.yml to reflect the naming changes.

Explore other deployment options & technologies & orchestrators (+supportive tooling for each), and define deployment recipes for:

• B2SHARE on OpenStack
  • Terraform, Openstack Heat, etc.
• B2SHARE on Kubernetes
  • ??
• B2SHARE on OpenShift
  • ??

[hjhsalo]

Architecture overview:

• Load Balancer
  • HAProxy?
• Frontend
  • Basically NGINX or other reverse-proxy
• REST API / Web worker
  • ??
• Background worker
  • Python Celery
• Database
  • PostgeSQL since this is the only thing we support in B2SHARE
• Elasticsearch
• Message queue
  • Currently RabbitMQ, and most probably this won't change. (Not even sure if anything else is supported)
• Redis
  • Used for caching in Invenio

Flower seems to be recommended by Invenio for monitoring of background worker(s): https://github.com/inveniosoftware/invenio-app-ils/blob/master/docker-services.yml#L84

[hjhsalo]

Open questions:

• Caching? What, when and at Load Balancer level or Frontend level?
• File storage options?
  • Currently B2SHARE stores files (files uploaded to drafts and records) on filesystem.
  • There is now Invenio-S3 as well (https://github.com/inveniosoftware/invenio-s3) that enables storing Object Storage using S3 API
• Configuration (and secrets) management?
  • B2SHARE can be configured via configuration file (https://github.com/EUDAT-B2SHARE/b2share/blob/master/b2share/config.py) and environment variables.
  • https://github.com/EUDAT-Training/B2SHARE-Training/blob/master/deploy/05_B2SHARE_configuration.md
  • https://github.com/EUDAT-Training/B2SHARE-Training/blob/master/deploy/04_Installation.md#set-environment-variables
• PostgreSQL scaling?