alexellis
commented
5 years ago Email from Wally from the NATS team / Apcera:
Hi Alex, you have a few options for HA NATS Streaming on K8S
to choose from depending on what you think is more flexible and
fits the use case better:
1) Multiple node Raft based cluster on ephemeral container filesystem
2) Multiple node Raft based cluster backed by persistent filesystem
3) Single instance backed up persistent filesystem
4) Single instance backed up by a DB store
The NATS Streaming operator (https://github.com/nats-io/nats-streaming-operator)
has support for all these options although technically you could do 3) and 4)
with a StatefulSet/Deployment with a single replica but the operator also provides
you with a CRD that you could can use if you want and some helpers for the configuration.
For the options 1) and 2) the operator really helps here since it
ensures that a single instance is ever defined with the `--cluster_bootstrap` flag
when creating the cluster for which there are no good workarounds
using StatefulSet/Deployments. 1) is what I think is the most
flexible since it does not have extra dependencies on a file system
but also means that if all instances from the clusterfail then there will be data loss,
but this could be mitigated by defining an anti affinity for the pods so that they are not
in the same host for example.
You can find some examples in the NATS Streaming repo here:
https://github.com/nats-io/nats-streaming-operator/tree/master/deploy/examples
Thanks,
- Wally
Production deployments of OpenFaaS should use an HA and Durable NATS installation to provide better stability. We should document how to install OpenFaaS and supporting an HA NATS cluster.