search

hetznercloud / hcloud-cloud-controller-manager

Kubernetes cloud-controller-manager for Hetzner Cloud
Apache License 2.0
711 stars 116 forks source link

clarification for ccm with networking support #297

Closed martyrs closed 1 year ago

martyrs commented 2 years ago

Hello, After reading a lot of documents about setting up a k8s cluster in hcloud, there is no clear explanation regarding CCM with or without networking support. I am aware that documentation is getting better over time but I am looking for clarification. My goal is to use Rancher and have already done a lot of experiments and I can say that it works fine. However...

I know every case and setup differ but my goal is to get most of out it while setting up my cluster. Thank you.

madalinignisca commented 2 years ago

I am following the md doc for network deployment and seems that cilium today evolved and creates on nodes and a new eth device created on one node will overlap the nodes private subnet, making the node unusable. I tried setting different values but can't figure out.

I think an Hetzner engineer should keep the doc updated with cilium changes as well, as the doc favorited it.

martyrs commented 2 years ago

I wasn't eager to test each case but CCM works fine with Calico on my end. To verify CCM is operational, simply issue kubectl get nodes -o wide and look for external/internal IP. Hope that helps.

madalinignisca commented 2 years ago

@martyrs, would you be kind to provide a hint on how did you setup calico? What values did you change for it's deployment?

martyrs commented 2 years ago

@madalinignisca sure, I use Rancher for cluster deployments. When creating your cluster,

madalinignisca commented 2 years ago

@martyrs that is what I'm still confused on. You are installing Calico with defaults that Rancher provides. Calico, Cilium, Flannel have different network backends, and each automation solution like Rancher has a default with settings that work on "bare-metal" setups. Check between PODS network traffic and you will see.

This does not means that the above setup will use Hetzner's network feature from the controller to use the private network for PODS networking, but you are using a network overlay on top of the public network (Calico will default it's default network to pass traffic through the default interface).

The goal is to have Hetzner CCM create the correct routes in the assigned private network, and the CNI to do the rest of setup.

I did manage to get with Kubernetes 1.22.x and Cilium 1.10.x all working without any issues using the instructions on https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/docs/deploy_with_networks.md and PODS and to SERVICES traffic all goes through the private network, but on 1.24, 1.25, with latest Cilium, CCM is not setting routes anymore. I will check the logs, but for the next couple of months, I'm sticking to older versions to get my project LIVE. After that, I hope I can contribute, as I will need also to upgrade in next <1y.

martyrs commented 2 years ago

@madalinignisca how exactly do you confirm your network topology? did you check the "routes" of your private network in hcloud console? after a working CCM installation, routes will be created there.

madalinignisca commented 2 years ago

@martyrs, so on your setup, you found the routes there? (I also did on 1.22.x and Cilium 1.10.x). Next, I'm checking by creating 2 pods (one with nginx and one with busybox) and test from busybox using traceroute. With the old version, I see them talking through the private lan. Later, don't work. My issue might be only that using latest Kubernetes and Cilium, is not working anymore.

PS: I deploy Kubernetes using kubeadm, I'm trying a more basic solution for my project, avoiding opinionated solutions like Rancher, K3s, Microk8s...

martyrs commented 2 years ago

@madalinignisca yes, routes there, CCM handles that routing part. Logs from a working CCM in action should be looking like this:

$ k logs deployment.apps/hcloud-cloud-controller-manager -n kube-system -f
....
I0826 09:03:46.335258       1 node_controller.go:325] Initializing node master1 with cloud provider
I0826 09:03:46.484193       1 route_controller.go:193] Creating route for node worker1 10.42.1.0/24 with hint 826ac9f0-1d50-4b84-9c2d-ff2a202e67ad, throttled 607ns
I0826 09:03:46.484277       1 route_controller.go:193] Creating route for node master1 10.42.0.0/24 with hint bc4a23e2-633d-4e24-b14c-c8d1b55fa95c, throttled 1.94µs
I0826 09:03:47.657836       1 node_controller.go:397] Successfully initialized node master1 with cloud provider

Never tested kubeadm way on Hetzner but I'll test it later on. Do you mind sharing your exact steps? that would be useful to reproduce.

madalinignisca commented 2 years ago

@martyrs my steps are the ones from https://v1-22.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ and https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/docs/deploy_with_networks.md I start with containerd setup, following the kubeadm, kubelet and kubectl setup (I do manual deployment, no deb packages), init the cluster, add working nodes, deploy ccm and then cni (cillium), setting tunnel: disabled and nativeRoutingCIDR: 10.244.0.0/16

martyrs commented 2 years ago

@madalinignisca I guess you are talking about that such pod-to-pod route takes place like this:

/ # traceroute 10.42.235.137
traceroute to 10.42.235.137 (10.42.235.137), 30 hops max, 46 byte packets
 1  static.0.xx.88.23.clients.your-server.de (23.88.xx.0)  0.014 ms  0.012 ms  0.009 ms
 2  *  *  *
 3  10.42.235.137 (10.42.235.137)  1.248 ms  0.765 ms  0.524 ms

It goes to public net first then returns to the node. Is this correct?

madalinignisca commented 2 years ago

yes. so, you have an overlay network, mapped throughout the public interfaces. to benefit of best performance and isolation, all must go throughout the private interfaces.

martyrs commented 2 years ago

Upon quick testing, I can confirm that traceroute (ICMP) not taking place on the public interface. You can just run tcpdump -i eth0 -s0 -vvv -n icmp then run traceroute "between" pods, you will notice it's not going out. Then, repeat same traceroute for outer world, like 8.8.8.8 and you will notice it's going out this time.

I will update this with more information as soon as possible.

madalinignisca commented 2 years ago

But why first hop is the public ip? I was expecting to see the private ip.

martyrs commented 2 years ago

I did more experiments and I can say that packets are never going to public net from my understanding. I believe the first hop is just fine and acts as the "router" part there. As CCM already managed and created "routes" in private networking, the packet gets to its destination internally.

Additionally, I can easily say that you will have serious latency issues without CCM, I had this until setting up CCM properly. Well, that's why we need clarification from Hetzner engineers in the first place :) Maybe @LKaemmerling and @fhofherr shed some light here.

madalinignisca commented 2 years ago

@madalinignisca I guess you are talking about that such pod-to-pod route takes place like this:

/ # traceroute 10.42.235.137
traceroute to 10.42.235.137 (10.42.235.137), 30 hops max, 46 byte packets
 1  static.0.xx.88.23.clients.your-server.de (23.88.xx.0)  0.014 ms  0.012 ms  0.009 ms
 2  *  *  *
 3  10.42.235.137 (10.42.235.137)  1.248 ms  0.765 ms  0.524 ms

It goes to public net first then returns to the node. Is this correct?

Sorted:

/ # traceroute 10.244.2.102 traceroute to 10.244.2.102 (10.244.2.102), 30 hops max, 46 byte packets 1 2 10.0.0.1 (10.0.0.1) 3.567 ms 3.569 ms 2.864 ms 3 4 10.244.2.102 (10.244.2.102) 1.116 ms 0.796 ms 0.336 ms

edit: Using cilium from e2e tests.

All traffic is now for me by private network isolated.

martyrs commented 2 years ago

@madalinignisca glad it worked for you. mind to share your approach and setup process? thanks.

madalinignisca commented 2 years ago

I'll take all notes, changes and make a blog post soon about it. I hope now I can get closer to "the minimal k8s cluster on hetzner cloud".

Basically:

  1. setup a network 10.0.0.0/8 with defaults (like hetzner panel does).
  2. setup a few servers (I start with 1 control-plane and 2 nodes), attaching them to the network (I did setup a snapshot image with common OS setup before, so next steps I usually skip)
  3. setup containerd (I do the manual setup, and all all container network plugins and rest of deps)
  4. setup kubeadm, kubelet, kubectl on latest or a maintained version as per kubernetes.io docs (I do again the manual setup)
  5. Edit on all servers /etc/hosts and add all nodes -- make sure to match the name in hetzner!!! (not necessary if you are going to do a dns server accessible in your private network)
  6. I'm adding a load balancer ip identical with first control-plane to be able to run the init, so later I can add the real load balancer and point to the new ip. Add it to hosts in all nodes.
  7. Example of init the cluster: kubeadm init --control-plane-endpoint=cplb.dev.saasified.dev --upload-certs --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.0.0.2 -v=5
  8. Add next the working nodes.
  9. Setup ccm with networks
  10. Apply cilium from this repo's e2e tests. This worked for me.

There is one thing confusing. The doc file for networks sais to match nativeRoutingCIDR to the cluster's subnet, but the e2e version is set to 10.0.0.0/8.

The result was routes as expected, Nodes got each correct ip pods pools allocated, things seem to work OK.

github-actions[bot] commented 1 year ago

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.