Docker improvements

[daniviga]

After #87 we have a multi-container version of the Engine.

Some work still need to be done.

Potential users are:

• Us to test cluster deployment with and without shared_dir
• Us to make our HPC clusters more flexible and more resilient to OOM
• BC Canada
• Suramericana
• maybe others....

[daniviga]

Host configuration should be passed at runtime (i.e. RabbitMQ hostname)

Building a Docker image is easy and rebuilding the -master and -worker (needed to change the openquake.cfg to point to differente hostnames) package is very quick.

[daniviga]

Persistent volumes (via -v or volumes)

On master a persistent volume is created either using -v to bind a local folder (es. for oqdata) or using volumes to use a Docker volume. Need just to document it.

[daniviga]

Shared volumes

On a single HW node this is useless because the engine container can be used (unless a cluster installation is used for testing), since it already uses all the cores.

On a multi-node HW extra tools, like GlusterFS or https://clusterhq.com/flocker/introduction/ or https://github.com/libopenstorage/openstorage or an NFS managed by the host system (and then shared to the container via -v) are required

[daniviga]

Docker cluster

Standards to manage a multi-node Docker cluster are

• Kubernetes
  • Swarm

We need to provide guidelines for them (and they are tending)

[daniviga]

Celery workers

Am env variable should be passed to the workers to limit the number of celery workers spawn (default is = to available CPU cores/threads)

[daniviga]

Done