search

sergelogvinov / proxmox-cloud-controller-manager

Kubernetes cloud controller manager for Proxmox
Apache License 2.0
124 stars 16 forks source link

Interaction with Sidero #45

Closed dhess closed 1 month ago

dhess commented 1 year ago

Hi, thanks for this project, it looks like it might be a great fit for our use case (also running Kubernetes clusters on a Proxmox cluster).

As background, we’ve just gotten a CAPI management cluster up and running via the Talos bootstrap, control plane, and infrastructure CAPI providers (the latter via Sidero). To create a new cluster, we currently need to:

  1. Manually create 1 VM for each node in the new cluster, spreading them around the various Proxmox nodes for HA.
  2. Generate a new cluster using clusterctl via the Sidero provider.
  3. Boot each node and let Sidero do its thing.

This works well enough, but it sounds like this CCM, combined with https://github.com/CRASH-Tech/proxmox-operator to create the QEMU VMs via a manifest, could eliminate at least step 1 above.

We would like for the new cluster procedure to be something like:

  1. Provision VMs in the Proxmox cluster for new Kubernetes nodes using Qemu CRDs via proxmox-operator.
  2. Generate a new cluster using clusterctl via the Sidero provider.
  3. Boot each node and let Sidero do its thing.
  4. Use the node labels generated by the Proxmox CCM, combined with your Proxmox CSI, to create node-locked PVCs for the services running on the new cluster.

Sorry for the dumb question, as I’m just figuring this stuff out, but as I understand it both Sidero and this CCM will be Node providers, so it’s not clear whether this will work. Will Sidero and this CCM both be able to label the nodes?

sergelogvinov commented 1 year ago

Hello, i did not try Sidero, i cannot say how it works. But i see here two ways to solve this idea.

  1. use Proxmox CCM in cloud-node-lifecycle mode only. It will delete the kubernetes node resource if VM was deleted. Sidero will promote and labels the nodes (providerID also required), Proxmox CSI is required topology.kubernetes.io/zone == $ProxmoxNodeName
  2. disable in Sidero to label the nodes, and use TalosCCM+ProxmoxCCM+ProxmoxCSI. TalosCCM in cloud-node (default mode), ProxmoxCCM in cloud-node-lifecycle mode.

If you familiar with terraform, there are two examples exist https://github.com/kubebn/talos-proxmox-kaas (good documented) and https://github.com/sergelogvinov/terraform-talos/tree/main/proxmox

dhess commented 1 year ago

Yes, I’ve seen https://github.com/kubebn/talos-proxmox-kaas and it even uses Sidero and CAPI, it’s just not clear to me how it works with the Proxmox CCM as well :)

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 14 days.

github-actions[bot] commented 1 month ago

This issue was closed because it has been stalled for 14 days with no activity.