OpenShift HA design

[smarterclayton]

OpenShift masters should be configurable to run in a redundant, failure tolerant, and horizontal scale out manner.

Masters are composed of three components:

• apiserver, which should be stateless and horizontally scalable in order to service cluster state change requests
• proxy bastions, which support high bandwidth proxy connections from clients to containers or nodes for port forwarding, remote execution, and log retrieval, and may demand a larger share of the masters bandwidth
• controllers, which generally expect to be the single writer for a particular control loop on the masters.

In the future, the master should be decomposeable into those three roles and scaled independently.

For an HA setup, the apiserver and proxy bastions should be run with more than a single instance, load balanced at an endpoint that is signed by the masters signing authority, for use by nodes and controllers (internal clients) and web and command line tools (external clients). The controllers should run a single instance, and if that instance stops running, another instance should be started.

Design points:

• the master internal and external endpoint names must be stable so they can be signed. Internally, they should be DNS names provided by the cluster
• controllers should leverage the etcd store to acquire a leader lease and run - when they lose the lease or die, they should terminate and restart into a candidate role. No external coordination should be required.
• the controller role should be easy to run independent of the apiserver and proxy bastion
• where possible, internal infrastructure should be used to run containers, rather than reimplementing the full OpenShift/Kube container stack

  Recommended maximum scale topology:

Run a singleton OpenShift master (the bootstrapper) managing a small number of infrastructure nodes, that hosts a set of masters and other infrastructure. The bootstrapper runs a replica set of masters and etcd nodes, offers persistent storage for etcd, manages health checks on the masters, and when the masters terminate, restarts them. The bootstrapper may leverage secrets to store the config for the hosted masters. The internal and external endpoints for the hosted masters should leverage internal and external service DNS names and the NodePort function to yield stable DNS. Other infrastructure components may be hosted with the master.

• One bootstrapper all-in-one system at a stable DNS
• N infrastructure nodes hosting client masters and other tools, talking to the bootstrapper
• 2+ instances of master, 3 instances of etcd running on infrastructure nodes
• M client nodes, pointing to client masters over a NodePort or LB backed by the client masters

Advantages:

• Masters are HA and reuse core tools
• More nodes required
• Isolate multiple installations via namespaces
• easy testing of failover

Disadvantages:

• Bootstrapper must be able to be restarted if it crashes

  Minimal node configuration:

Run three (or more) openshift infrastructure nodes, with local file manifests for each instance of the master, and create similar for the etcd cluster. Use a local host directory to store the master and etcd config and etcd storage, and have the masters listen on the host network. Configure each infrastructure node to loop back to the current host to connect to the master on the external port. Each node should then be configured to point to a DNS address containing the master infrastructure node. Optionally, the infrastructure nodes can use that DNS address.

Disadvantages:

• requires external load balancer to get HA (without assuming fast DNS failover)

TBD:

• describe configuration of each set of containers

[smarterclayton]

@liggitt @danmcp @detiber @brenton @twiest @derekwaynecarr @ncdc

This is a draft of the long term HA design. It's a continuation of the Kube HA design but pushes it into self hosting and bootstrapping - we're likely to discuss this upstream post 1.0, but I'd rather not spend the next year reinventing the wheel for our core stories.

[detiber]

@smarterclayton I like the idea of bootstrapper hosts, but I dislike the idea of it being a singleton host, we'll need some type of HA story for it, even if it is just RHEL HA in an active/passive manner.

[smarterclayton]

Why do we need an HA story for it? Why can't the HA story be "restart the machine"? Why is active/passive better than "get an instance running again with the same DNS and disks"?

On Mon, Jul 13, 2015 at 11:03 AM, Jason DeTiberus notifications@github.com wrote:

@smarterclayton https://github.com/smarterclayton I like the idea of bootstrapper hosts, but I dislike the idea of it being a singleton host, we'll need some type of HA story for it, even if it is just RHEL HA in an active/passive manner.

— Reply to this email directly or view it on GitHub https://github.com/openshift/origin/issues/3597#issuecomment-120960544.

Clayton Coleman | Lead Engineer, OpenShift

[smarterclayton]

This is implemented @sdodson also FYI

[abhat]

@smarterclayton is this already in v3.1?

[smarterclayton]

This is in origin 1.0.6, and will be supported by the ansible installer for origin 1.1 and OSE 3.1