Expose more StatefulSet+Pod+Container functionality through Habitat CRD

[HT154]

The Habitat operator is great for running simple services, but for services with more advanced requirements, it can be rather limiting. Here are some features of Pods that the Habitat CRD should consider exposing:

• affinity
• containers - in addition to the primary operator-managed habitat-service container (e.g. running a Splunk forwarder)
• dnsConfig, dnsPolicy, hostAliases, subdomain
• initContainers
• nodeSelector, tolerations
• securityContext - particularly for setting fsGroup: 42 (the hab user's gid) to fix persistent volume permissions issues
• serviceAccountName - potentially useful for running operators under the habitat operator
• volumes - in addition to the volumes configured by the operator (e.g. for sharing a logs directory with a Splunk forwarder sidecar)

Properties on the habitat-service Container that should be adjustable:

• lifecycle, livenessProbe, readinessProbe
• resources - required for running in clusters that enforce pod resource limits/requests
• securityContext
• volumeMounts - in addition to operator-configured mounts, enables aforementioned Splunk use case

Properties on the StatefulSet that should be exposed:

• podManagementPolicy
• serviceName
• volumeClaimTemplates - in addition to operator-configured PVC templates

[irvingpop]

Great list - the lack of liveness & readyness probes is a bummer. I've also found myself wishing that the operator would expose topology type and leader election status.

[HT154]

+1 On leader election status. It'd be great to set up a Service with a selector that selects for only the leader of a group. I was just wishing for this the other day for setting up redis replication.

[HT154]

Updated original issue to remove StatefulSet updateStrategy from consideration. This is used internally by the operator to manage deleting pods and probably shouldn't be overridden.