Configure RabbitMQ for durable queues

[gothub]

Currently the RabbitMQ server used by metadig-engine is not configured for durable queues. With durable queues, queue entries can be written to a persistent volume so that if the pod is restarted, the queue entries are re-read and not lost.

Tasks required for durable queues:

• set queues and exchanges as 'durable', and messages as 'persistent'
• convert the RabbitMQ helm chart from using Deployment to StatefulSet
  • current chart: https://github.com/NCEAS/metadig-engine/blob/feature-helm-%23152/helm/rabbitmq/templates/deployment.yaml
  • this is needed as the PV directory includes a node identifier, which a StatefulSet provides
  • this is also required for a RabbitMQ cluster
• determine which type of PV to use
  • this can be different than the PV used by other metadig-engine services

[gothub]

RabbitMQ from Bitnami is now running on dev k8s as a 3 node cluster:

kubectl get pods -n metadig -o wide 
NAME                                      READY   STATUS              RESTARTS   AGE    IP               NODE                NOMINATED NODE   READINESS GATES
pod/metadig-rabbitmq-0                    1/1     Running             0          4m5s   192.168.3.179    docker-dev-ucsb-1   <none>           <none>
pod/metadig-rabbitmq-1                    1/1     Running             0          4m5s   192.168.50.181   docker-dev-ucsb-2   <none>           <none>
pod/metadig-rabbitmq-2                    1/1     Running   0          6m      192.168.3.134    docker-dev-ucsb-1   <none>           <none>

kubectl get services -n metadig -o wide
NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                         AGE   SELECTOR
service/metadig-rabbitmq            NodePort    10.104.93.246   <none>        5672:32483/TCP,4369:32722/TCP,25672:30721/TCP,15672:31302/TCP   6m    app.kubernetes.io/instance=metadig-rabbitmq,app.kubernetes.io/name=rabbitmq
service/metadig-rabbitmq-headless   ClusterIP   None            <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP                           6m    app.kubernetes.io/instance=metadig-rabbitmq,app.kubernetes.io/name=rabbitmq

Metadig-engine has been refactored to use durable queues and dynamically provisioned persistent storage. Metadig-engine queue requests are retained if metadig-rabbitmq is stopped and restarted, or if one of the pods is deleted (simulating a node restart).

This functionality will be installed on production k8s in the metadig-engine 2.4.0 release.