Rethink CRD

[krnowak]

Issues like #264 and #345 are showing that there is a need to expose more and more {StatefulSet,Pod,Container}-specific settings in the CRD. These currently are not configurable at all, because whatever change a user makes to the {StatefulSet,Pod,Container} created by the operator, that change will be clobbered immediately by the operator. That's even stated in the design document - all subresources created by the operator should be treated as readonly for the user.

So I had the idea that the Habitat CRD could basically use a PodTemplate in its CRD, so using it would be a kinda similar experience to using Deployment or Statefulset. Now, whether we use PodTemplateSpec or a sizable subset of it needs more thought. Using PodTemplateSpec directly is faster to write and easier to maintain. OTOH, that would expose the possibility to specify the arguments to be passed to the supervisor and this functionality is what most likely the operator would want to be in full control of.

Using PodTemplateSpec (or its subset) wouldn't allow exposing the functionality of StatefulSets to the user. I'm not sure what to do in this case. Using StatefulSetSpec would only be acceptable if we decided that both using PodTemplateSpec directly and making StatefulSet not an implementation detail of Habitat CRD are acceptable.

[jtimberman]

@krnowak Would this allow specifying multiple containers in the spec for Habitat resources? For example, a Pod Template has containers, which can specify multiple containers (in the use case of running a sidecar or background worker container in the pod). This doesn't seem possible with the current Habitat resource, however.

[jkerry]

This would be nice. The lack of readinessProbe capability is hurting my k8s habitat POC in a bad way.