search

syself / cluster-api-provider-hetzner

Cluster API Provider Hetzner 🚀 Kubernetes Infrastructure as Software 🔧 Terraform/Kubespray/kOps alternative for running Kubernetes on Hetzner
https://caph.syself.com
Apache License 2.0
543 stars 51 forks source link

Use LoadBalancer IPv6 address #1218

Open JochemTSR opened 4 months ago

JochemTSR commented 4 months ago

/kind feature

Describe the solution you'd like Currently, the Kubernetes API server endpoint is always set to the load balancer's IPv4 address. I'm setting up a Hetzner cluster with IPv6-only nodes and a private network; all incoming traffic would be handled by Hetzner load balancers. As the API server endpoint is set to an IPv4 address, nodes cannot reach the API server after kubeadm initialization (dial tcp 167.xxx.xxx.xxx: connect: network is unreachable).

As far as I can tell, a pure IPv6 setup is currently not possible without manually configuring a load balancer and DNS so a hostname can be used for spec.controlPlaneEndpoint.host in the HetznerCluster manifest.

Looking at the code that determines the endpoint, I think it should be possible to add a property to the controlPlaneLoadBalancer object (e.g. useV6Endpoint). If this property were set to true, the control plane endpoint would be set to the LB's IPv6 address instead of the IPv4 one in the code section below:

https://github.com/syself/cluster-api-provider-hetzner/blob/3bc16ac0c7fa2273ff693f21a55bf1d193efab4a/controllers/hetznercluster_controller.go#L244-L265

Please let me know what you think, I'd be happy to contribute a PR if this seems feasible/desirable.

Anything else you would like to add: Related issues: #632, #899

Environment:

guettli commented 4 months ago

@JochemTSR thank you for your feedback. Up to now we (Syself) and users of our commercial offering are happy with ipv4 only.

Just out of curiosity: Why do you want to use ipv6? What is the benefit, what is your goal?

During the next weeks we are not able to work on that. Maybe later.

But feel free to create PR which works for you.

Ping me if you have difficulties to create a PR and test it in your cluster.

If you want ipv6 for your applications running inside the caph cluster, then this is a different topic.

JochemTSR commented 3 months ago

@guettli Apart from IPv6 being The Future™, the main benefit would be the 15% discount on the CAX11 nodes for not using a public IPv4 address. Provisioning nodes without IPv4 is already possible with the Hetzner provider by setting enableIPv4 to false in the machine template.

For me personally it's mostly a fun exercise in setting up my hobby cluster.

JochemTSR commented 3 months ago

I should add, the cluster I'm envisioning does use IPv4 for the internal network, mostly because this is what the Hetzner private networks offer.

guettli commented 3 months ago

@guettli Apart from IPv6 being The Future™, the main benefit would be the 15% discount on the CAX11 nodes for not using a public IPv4 address.

Wait a second. CAX11 are ARM machines. Do you run caph on arm?

Thank you for this hint. This makes sense. If it is a hobby project for you, then why not dive into the big fun create a patch to make it working?

apricote commented 3 months ago

Wait a second. CAX11 are ARM machines. Do you run caph on arm?

Off topic: I do run CAPH on the Arm server types since they launched and it works great 💪

JochemTSR commented 3 months ago

@guettli It took some fiddling to get the network to play nice, but I managed to provision a functional cluster with IPv6 only. The development tooling of this project is actually quite nice once you get familiar with it :)

On a related note, I think the development documentation is a bit out of date? It's no biggie but it did take a bit longer to get started. Is that worth opening a seperate issue for?

janiskemper commented 3 months ago

@kranurag7 can you maybe have a look at the development documentation and open an issue? I have also seen that a few days ago but forgot to open an issue. We should really update it!

kranurag7 commented 3 months ago

On a related note, I think the development documentation is a bit out of date? It's no biggie but it did take a bit longer to get started. Is that worth opening a seperate issue for?

@JochemTSR If you intend to open an issue then please go ahead and let us know what can be improved here. If not, I'll open up an issue and fix the same. Please let me know if you've some specific feedback.

janiskemper commented 3 months ago

@kranurag7 it is really outdated. Just open the issue because you know the setup. @JochemTSR can add something on top if he doesn't find his topic in your issue!

JochemTSR commented 3 months ago

@kranurag7 @janiskemper opened #1233 to track possible improvements to the development documentation.

janiskemper commented 3 months ago

thanks a lot @JochemTSR !