[FR] - Introduce more control nodes, and turn all rack-mounted servers into workers

[whit-colm]

Define the scope of the new feature 3 new machines should be procured and introduced to the cluster to act as control nodes, Formula, the current single control node should recycled as a worker. The new nodes in turn will allow for proper high availability and decrease resources that could be used better elsewhere.

Why is this needed?

• There is a single point of failure within the cluster, while any of the 3 worker nodes can fail if the single controller fails the cluster goes down
• A single control plane makes upgrading Talos a kinda dirty affair. Upgrades wipes the EPHEMERAL partition by default, which contains the etc.d data among other critical components. For a single controller, that means the EPHEMERAL partition needs to be retained indefinitely, rather than a regular complete re-instantiation from the upgraded controller's peers.
• The power used by formula and the resources it has are wasted on acting as a control plane.
  • (From Prometheus) Out of ~62GiB allocable memory, 2.1GiB is requested and 3GiB actually used.
  • (From Prometheus) Out of 32.00 allocable CPU, 0.96 is requested and 0.59 actually used.
  • (From HP iLO) 82W of power is used by the machine.

What needs to happen first?

• [ ] Understand the necessary hardware requirements for 3 control plane nodes
  • Assuming eventually 6 worker nodes will be managed.
• [ ] Find affordable hardware fitting the above.

[whit-colm]

When implementing, remember to add the necessary IPs to the static configs so kube-prometheus-stack can scrape resource usage data.

[whit-colm]

There is zero way I am getting three machines in line by 2023-08-16. I'm repurposing an old laptop to serve as indefinite controller until that point. It won't be pretty, but it's the only way this project has a whelk's chance in a supernova of going to schedule.