Add Tolerations to cluster and node CRDs

[JohnStrunk]

Describe the feature you'd like to have. We currently support specifying node affinity to constrain where pods are placed. We should also support Tolerations so that our pods can be placed on dedicated nodes if desired.

What is the value to the end user? (why is it a priority?) An admin may want to have dedicated kube worker nodes for the gluster storage server pods. This requires 2 things to happen:

1. non-gluster pods must be prohibited from running there
2. gluster pods should be directed to these nodes

In order to implement this, the admin would apply a 'noschedule' Taint to the designated nodes, preventing pods from being scheduled there, effectively implementing (1). To achieve (2), it requires both the existing NodeAffinity to direct the pods to these nodes and specifying a toleration on the pods to permit them to ignore the taint.

How will we know we have a good solution? (acceptance criteria) It should be possible to run gluster pods on a dedicated set of nodes via the combination of taint/toleration and node affinity.

Additional context This requirement arose based on a discussion of how machinesets will be used with workloads requiring dedicated resources in OpenShift.

Major work items:

• Revise CRDs to support tolerations (adding a []*corev1.Toleration to both CRDs)
• Code to take the toleration from the GlusterNode and add it to the gd2 StatefulSet during reconcile

[jarrpa]

What specifically about MachineSets affected this decision?

[JohnStrunk]

It's not tied to the MachineSet method of scaling per se. It's that affinity gives only 1/2 the solution-- taints/tolerations are required also. The discussion is just what made the need apparent.