Containers: Compute to Rabbit Communication

[bdevcich]

From https://github.com/NearNodeFlash/NearNodeFlash.github.io/tree/containers-communication-update/docs/rfcs/0002#compute-to-rabbit-communication:

Compute-to-Rabbit Communication

For Compute to Rabbit communication, the proposal is to use an open port between the nodes, so the applications could communicate using IP protocol. The port number would be assigned by the Rabbit software and included in the workflow resource's environmental variables after the Setup state (similar to workflow name & namespace). Flux should provide the port number to the compute application via an environmental variable or command line argument. The containerized application would always see the same port number using the hostPort/containerPort mapping functionality included in Kubernetes. To clarify, the Rabbit software is picking and managing the ports picked for hostPort.

This requires a range of ports to be open in the firewall configuration and specified in the rabbit system configuration. The fewer the number of ports available increases the chances of a port reservation conflict that would fail a workflow.

[bdevcich]

The ports for all the rabbit containers should be the same for a workflow. Then, the compute nodes just need to know how to contact their local rabbit. One potential solution is to have site admins add a dns entry on each compute node for rabbit.local that resolves to the connected rabbit. Then rabbit-local:PORT would be a simple way for a compute node application to communicate with the container running on the rabbit.

[bdevcich]

Manual ports can be opened by supplementing the worker PodSpec in a container profile with ports and by using a hard-coded value here and on the compute node.

  containers:
  - name: example-mpi-webserver
    ports:
    - containerPort: 2000
      hostPort: 2000

Since these ports are opened on the host (i.e. NNF node), only 1 port can be open at a time, which means that only 1 container workflow can be active per container profile with the manual approach.

Once the container is running on the NNF node, the compute nodes can hit the port via <NNF_NODE_IP>:<PORT>.

The IP of the NNF node can be obtained by running kubectl get nodes -o wide. For the full solution, it is recommended that each compute node contains a local rabbit host in /etc/hosts to make these easy for applications running on the compute node.