search

vitobotta / hetzner-k3s

The easiest and fastest way to create and manage Kubernetes clusters in Hetzner Cloud using the lightweight distribution k3s by Rancher.
MIT License
1.9k stars 141 forks source link

Feat: Proper use of the Hetzner private network to minimize latency and remove the need for encryption. #135

Closed aleksasiriski closed 1 year ago

aleksasiriski commented 1 year ago

If I'm not mistaken, this script uses flannel with wireguard?

Hetzner Cloud Controller Manager allows for this functionality.

vitobotta commented 1 year ago

Yes, depending on the k3s version selected it will use the new wireguard-native support or the old wireguard method.

vitobotta commented 1 year ago

Just to clarify: the enable_encryption config setting is what determines whether or not to use wireguard.

aleksasiriski commented 1 year ago

Just to clarify: the enable_encryption config setting is what determines whether or not to use wireguard.

Wouldn't it be a much better default to not use encryption inside a private network / subnet for pods?

vitobotta commented 1 year ago

Just to clarify: the enable_encryption config setting is what determines whether or not to use wireguard.

Wouldn't it be a much better default to not use encryption inside a private network / subnet for pods?

The default is false, but I enable it usually because on this page https://docs.hetzner.com/cloud/networks/faq/ it says:

Is traffic inside Hetzner Cloud Networks encrypted? Traffic between cloud servers inside a Network is private and isolated, but not automatically encrypted. We recommend you use TLS or similar protocols to encrypt sensitive traffic.

vitobotta commented 1 year ago

@aleksasiriski I have released v.1.0.3 which allows using Rocky Linux etc, and now I am looking into MicroOS out of curiosity. I found a very easy and quick way to create a MicroOS snapshot with https://github.com/kube-hetzner/packer-hcloud-microos (takes just a couple of minutes to create a MicroOS snapshot), but I am having a couple of problems:

Would you have any suggestions as how to solve the above? I would like to make my tool compatible with MicroOS but I am not familiar with it and Google isn't helping much...

vitobotta commented 1 year ago

I haven't played with MicroOS, maybe try asking the author of terraform-hcloud-kube-hetzner

Where I could help and would also recommend is to maybe try Fedora CoreOS instead, since that is what Openshift uses for it's nodes. I would also help in this, might even learn some Crystal.

In the end, as I can see you're highly skilled and motivated I'll stay on this project for my cluster and try to bring more features here as well. It's rare to see someone so friendly and open to supporting features the community asks of an open source project, I am truly thankful for that!

I'm almost there with the MicroOS support :D Got a cluster fully working! Just need to adjust a few things to keep compatibility with other distros. I have the feeling that CoreOS might work as well after this, let's see.

vitobotta commented 1 year ago

Good news! I released v1.0.4 which makes it possible to use MicroOS with a custom snapshot 🎉 You can quickly create a snapshot using this tool https://github.com/kube-hetzner/packer-hcloud-microos - it takes a couple of minutes and it's only needed once per project. Then you set the image config option to the ID of the snapshot and that's it! I tried several clusters and they worked just fine, with one caveat: autoscaled node pools are not supported unfortunately. This is something I can't do much about because when a server is created from a snapshot, the normal cloud init process that is usually used to initialize a server is not triggered; when I create nodes in advance for non-autoscaled node pools, I can run SSH commands into the MicroOS servers and do some stuff to configure networking etc, but when the nodes are created by the autoscaler, which requires cloud init, these tasks cannot be performed and the autoscaled nodes cannot join the cluster. It's annoying but I can't do anything about it from what I can see. I will investigate more with snapshots later just in case I find a workaround. But with normal node pools MicroOS works just fine, so I guess it's an improvement.

Thank you for bringing up the topic of non-Ubuntu Linux distros because it helped me make the tool more flexible :)

aleksasiriski commented 1 year ago

Maybe the creating of snapshot needs to be integrated as well, so that the tool configures everything that it can over cloud-init and then uses that universally configured snapshot?

vitobotta commented 1 year ago

Maybe the creating of snapshot needs to be integrated as well, so that the tool configures everything that it can over cloud-init and then uses that universally configured snapshot?

I think I found a solution. Working on it.

vitobotta commented 1 year ago

Making progress! I released v1.0.5 which fixes support for autoscaling when using a MicroOS snapshot 🎉 So MicroOS is now fully working. :)

I'm gonna look into Fedora CoreOS next.

opsdevit-nl commented 1 year ago

Making progress! I released v1.0.5 which fixes support for autoscaling when using a MicroOS snapshot tada So MicroOS is now fully working. :)

I'm gonna look into Fedora CoreOS next.

That's really awesome! I'm looking forward to the release containing CoreOS!! Thanks a ton for all your effort and unstoppable pace!

aleksasiriski commented 1 year ago

Making progress! I released v1.0.5 which fixes support for autoscaling when using a MicroOS snapshot 🎉 So MicroOS is now fully working. :)

I'm gonna look into Fedora CoreOS next.

That's great to hear! Would you look into implementing kured since both MicroOS and CoreOS have automatic updates enabled by default?

vitobotta commented 1 year ago

Making progress! I released v1.0.5 which fixes support for autoscaling when using a MicroOS snapshot 🎉 So MicroOS is now fully working. :) I'm gonna look into Fedora CoreOS next.

That's great to hear! Would you look into implementing kured since both MicroOS and CoreOS have automatic updates enabled by default?

I'll look into it but it will likely be next weekend since my holiday is over and tomorrow I start working again. Unless I find the time one of these evenings

aleksasiriski commented 1 year ago

kube-hetzner makes use of Hetzner networks by creating subnets for every node so that pods have seperate isolated networks which alleviates the need of wireguard network. I'm just a beginner in Kubernetes but I believe this is a much better and less resource intensive approach?

vitobotta commented 1 year ago

kube-hetzner makes use of Hetzner networks by creating subnets for every node so that pods have seperate isolated networks which alleviates the need of wireguard network. I'm just a beginner in Kubernetes but I believe this is a much better and less resource intensive approach?

I am not sure I am following. Can you point me somewhere in their docs or code? I could not see anything about it in the README

aleksasiriski commented 1 year ago

Their README:

Features

Proper use of the Hetzner private network to minimize latency and remove the need for encryption.

I found this code.

aleksasiriski commented 1 year ago

When using kube-hetzner to create a cluster I noticed subnets being created in the cluster network on Hetzner Console (number of subnets equal to the number of nodes). I also noticed that on each node pods use those different subnets.

And according to this you actually can use that functionallity:

Features

Private Networks: allows to use Hetzner Cloud Private Networks for your pods traffic.

aleksasiriski commented 1 year ago

I just noticed that your scripts makes some routes that look similar to subnets, but I don't quite understand what these routes are and if they achieve the same goal?

vitobotta commented 1 year ago

Both tools do the same thing basically. The tools create a private network with a subnet, and then the Hetzner Cloud Controller Manager creates routes for each node.

aleksasiriski commented 1 year ago

Then I guess this is already implemented, but I would still argue that encryption is not needed as the default but it's not hurting much of the performance.

vitobotta commented 1 year ago

Yeah maybe it's just me being paranoid :) by default it's disabled anyway.

aleksasiriski commented 1 year ago

Yeah maybe it's just me being paranoid :) by default it's disabled anyway.

"By default" I meant in the example config, since it's not commented out.

vitobotta commented 1 year ago

Yeah maybe it's just me being paranoid :) by default it's disabled anyway.

"By default" I meant in the example config, since it's not commented out.

I have changed it :)