search

siderolabs / talos-vmtoolsd

VMware tools implementation for the Talos Kubernetes platform, using govmomi and Talos' apid
Apache License 2.0
28 stars 14 forks source link

Bootstrap Talos via CAPI #1

Closed dimatha closed 3 years ago

dimatha commented 3 years ago

First of all thanks for the great job !

As Cluster API relies on the IP address of the machine to be reported back to the vCenter I'm forced to use vmware tools and this is how I came across this project.

The problem with CAPI deployment is that I need to add additional logic to get the token from the talos config and push the secret to the target cluster (via CAPI ClusterResourceSet) to get the vmware pod started. I was just looking for some hints on how this part can be automated or done better when it come to CAPI provisioning.

Thanks!

mologie commented 3 years ago

Hi! Bootstraping talos-vmtoolsd is indeed rather awkward due to the dependency on a Talos API certificate. I agree that this is an issue, but I think I cannot fix it within talos-vmtoolsd alone.

This was discussed previously w/o referring to CAPI specifically in #proj-talos-vmware of the Talos Slack server, and we did come up with a possible design, but no solution thus far.

The general consensus was that tools such as talos-vmtoolsd, i.e. Talos extensions running in Talos under K8s, should be able to use some service which issues them short-lived and properly scoped Talos API tokens on demand or automatically as K8s secret. However, no such Talos to K8s integration exists at the time of writing (and any solution that does not come from Talos itself will too have this bootstrapping issue).

Alternatively I suggested to simply integrate talos-vmtoolsd into Talos' machined, because it includes govmomi for reading the config anyway. However, hypervisor integration was seen as out-of-scope for machined. Indeed an RPC based extension API is preferable.

I hear that the folks at Talos are investigating CAPI with Talos under vSphere and they are likely to run into the same issue when they need a VMware Tools implementation. I do not have a solution for you here, but maybe you can revive the discussion in #proj-talos-vmware; I idle there too and will likely see your message. I'd happily contribute changes to talos-vmtoolsd once an in-cluster token API or different solution that solves the bootstrapping issue becomes available.

dimatha commented 3 years ago

Thank you for the comprehensive answer !

I've also raised this question with Talos folks and they have something already ongoing here: https://github.com/talos-systems/talos/issues/4422

Have a great weekend !