Create clustered instances

[phlogistonjohn]

This is the preliminary set of patches to add clustering support, aka CTDB and StatefulSets, to the operator. This enables the designed, but so far unimplemented, scaling subsection to the smbshare CR.

It's already long and bound to get a tad bit longer. I just don't see any great places to split out more patches ATM.

What's added (so far)

• Updates to the CR
• Ability to configure CTDB enabled containers
• Ability to produce support state PVC
• Create the statefulset
• A test

Still pending (not a final/complete list)

• AD Enabled & CTDB Enabled instances
• Blocking casual use - adding a specific operator configuration option to opt in to the experimental ctdb support
• Does not deal with changes to the SmbShare w.r.t.. scaling
• Full headless service support
• LoadBalaner Service (access outside the k8s cluster)
• dns-reg support
• preventing/erroring on trying to switch between availability modes

Some of the pending items, specifically AD support and opt-in to ctdb should be added as part of this pr

[phlogistonjohn]

Hehe. I just realized the new test cannot pass the CI because it requires multiple nodes. I either need to disable the PodAntiAffinity (for our tests) or we need multi node k8s in the CI.

[phlogistonjohn]

The new code also requires RWX PVC support. Without trying to boil the ocean and re-work the test infra we'll skip the clustered instance testing unless explicitly requested.

[phlogistonjohn]

Based on the size of this PR, I don't intend to add any more changes here. There are a number of things left undone but are not huge gaps. Depending on quick reviews are this following list will either have Issues or PRs when this PR is merged.

Skipping for now:

• Specific headless service(s) for the stateful set
• Support (and tests) for loadbalancer service for stateful sets
• AD dns registration (probably needs changes in sambacc)
• Handling (explicitly blocking) changing between availability modes
• Proper support for scaling (up/down) the size of cluster/stateful set