Load balancer annotations.

[Aethrexal]

Edit: Made some parts easier to read.

This isn't an issue with this script or anything, it seems that I'm just very stupid 😓, I don't know where else to ask except here since using this script and easier point to go from.

So, I give up lol. I've looked through so many docs and guides and what not but, I'm so confused how I'm suppose to get rancher to work. I mean, rancher and cert-manager boots up and the pods gets ready, but the whole load balancers and that confuses the hell out of me. Load Balancers It's that part that confuses me a lot. Where I would place those and so on.

I thought I made progress when I found The load balancer README on hcloud CCM, but even that confuses me and I'm not sure how I'm suppose to set it all up going from this script. I do feel like I still made some progress, specially also using Lens as that made it a bit easier to have a look at everything going on with the cluster.

I assume the hostname would be load.example.com with setting up the DNS on cloudflare, with type A where load.example.com points to Load balancer Public IP. I might be wrong with even that.

So basically, would be forever grateful with some help and guidance with this. Cause either I'm missing something in all docs and guides I've been reading, or I can't read, or I'm simply just stupid lol.

[vitobotta]

Not sure I understand your questions 100% but let's try. So there's two things for which you need a load balancer. First, for the Kubernetes API server only if you create a cluster with multiple masters; in this case, you don't have to do anything as the installer automatically creates and configures a load balancer for the API server.

Then there are load balancers for your workloads. The installer installs the cloud controller manager that allows you to create Kubernetes services of type load balancer out of the box, but you need to specify at least two annotations: 1) one annotation which sets the Hezner location, otherwise the load balancer won't be provisioned and will stay in "pending" state.; 2) the annotation that ensures that the communication between the load balancer and the cluster nodes happens through the private network. This is to avoid opening additional ports on the nodes. So at a minimum when you create a service of type LoadBalancer you need these annotations:

  service:
    annotations:
      load-balancer.hetzner.cloud/location: nbg1
      load-balancer.hetzner.cloud/use-private-ip: "true"

The other annotations I mentioned in the README are only needed if you want your apps to "see" the real IP of the client, because otherwise they will see the IP of the load balancer. But I suggest you skip this for now until you are more familiar with this.

Since I only deal with regular web apps, what I usually do in my clusters is install the Nginx ingress controller and let Kubernetes create a load balancer for it. Then for each web app I create an ingress resource, so all of them will share a single load balancer in the end, which is cheaper and easier.

To easily install the Nginx ingress controller you can do this. First, run the following to create a yaml file with the values for the Nginx ingress Helm chart:

cat << EOF > /tmp/ingress-nginx.yaml
controller:
  kind: DaemonSet
  service:
    annotations:
      load-balancer.hetzner.cloud/location: nbg1
      load-balancer.hetzner.cloud/name: cluster-name-ingress-nginx
      load-balancer.hetzner.cloud/use-private-ip: "true"
EOF

And then install the Nginx ingress controller using that configuration:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install \
--namespace ingress-nginx \
--create-namespace \
-f /tmp/ingress-nginx.yaml \
ingress-nginx ingress-nginx/ingress-nginx

Since you've specified at least the Hetzner location, the cloud controller manager will correctly create a load balancer for Nginx. It usually takes a minute or so. You can check from the Hetzer console but it takes a 1-2 minutes for the load balancer to get the IP as seen by Kubernetes. You can check with kubectl:

kubectl -n ingress-nginx get svc

After one or two minutes the LoadBalancer service should no longer be "pending" and should have a public IP assigned to it. That means that you are ready to create ingress resources. So you can use that public IP to configure DNS records for your apps.

Now, look at this example deployment: https://gist.githubusercontent.com/vitobotta/6e73f724c5b94355ec21b9eee6f626f1/raw/3036d4c4283a08ab82b99fffea8df3dded1d1f78/deployment.yaml

As you can see there's a service for the deployment, and an ingress resource. The service is of type ClusterIP by default because we don't need a load balancer for it, since it will be sharing the load balancer created for the ingress controller (nginx) with other services.

Look at the ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
spec:
  rules:
  - host: hello-world.<load balancer ip>.nip.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80

Once you replace <load balancer ip> with the public IP assigned to the load balancer created automatically for Nginx, your sample deployment will be reachable at the url http://hello-world.load balancer-ip.nip.io - nip.io, if you have never heard of it, is just a quick way to test things without configuring DNS (a query to a hostname ending in nip.io simply returns the IP address it finds in the hostname itself).

So as you can see you only need to create a load balancer when you install the ingress controller. Nginx, and that load balancer is created automatically for you. So you really don't need to do anything else / you don't need to deal with load balancers anymore. Just create one ingress resource for each web app and you're done.

Now, if you want provision TLS certificates with cert manager, you first install the helm chart:

helm repo add jetstack  https://charts.jetstack.io

helm upgrade --install \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true \
cert-manager jetstack/cert-manager

Then you need to configure one or more issuers or cluster issuers which will take care of doing stuff to obtain certificates. For example for production certificates you would run this:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: your@email.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

This creates a cluster issuer that can be used in any namespace. All you need to do in order to trigger the issuance of a certificate is update the ingress resource for your app with two changes:

1. Add a couple of annotations
2. Configure TLS

So the ingress resource as in the same deployment I linked to earlier would become:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/tls-acme: "true"
spec:
  rules:
  - host: yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80
  tls:
  - hosts:
    - yourdomain.com
    secretName: yourdomain-tls

Ensure that before creating or updating the ingress resource, you have already configured the DNS so that yourdomain.com points to the IP of the Nginx load balancer.

Within a couple of minutes, the certificate is provisioned and the app should be reachable at https://yourdomain.com.

These are the basics. Like I mentioned in the REAME if you want your apps to see the client's real IP address you need a few more annotations on the Nginx service, which I explained in the README.

As for Rancher, once you have the ingress controller (Nginx) and cert-manager up and running, it's easy to install with Helm:

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
helm repo update

kubectl create namespace cattle-system

helm upgrade --install \
--namespace cattle-system \
--set hostname=rancher.yourdomain.com \
--set ingress.tls.source=letsEncrypt \
--set letsEncrypt.email=your@email.com \
rancher \
rancher-stable/rancher

With the commands above, and specifying your actual Rancher domain and your email address (for cert-manager notifications), Rancher should be up and running in a few minutes. Just ensure you configure the DNS for the Rancher domain in advance so that it points to the Nginx load balancer IP.

Wait until the deployment is ready (kubectl -n cattle-system rollout status deploy/rancher) and then Rancher should be reachable at https://rancher.yourdomain.com.

Remember to create a new admin and disable the default admin.

This should help you get started hopefully :)

[Aethrexal]

This is exactly what I needed! Thank you so much 😃

First, for the Kubernetes API server only if you create a cluster with multiple masters; in this case, you don't have to do anything as the installer automatically creates and configures a load balancer for the API server.

Then there are load balancers for your workloads.

This is one point where I was confused the most. Cause I thought the load balancer that got created automatically for the API was also the same that would be used for making DNS for apps. I just couldn't get it all to fit together reading all the different docs and watching different videos etc haha.

So I was just at a complete stand still, and this just made everything so much clearer for me. So hopefully it should all go much smoother for me now haha. Thanks again for the help!

[vitobotta]

Nope, the API server requires its own load balancer, that is unrelated to the load balancer used by the apps. Note that the installer only creates the LB for the API server if you specify more than 1 master, otherwise it doesn't and configures the kubeconfig to connect directly to the single master. Let me know if you have other questions otherwise I will close this. :)

[Aethrexal]

Oh make sense 😃

That would be all but, I might be missing something cause it's not creating the volumes.

1. I go to Volume tab to make a main volume (Don't want to create many small yet, specially for things I don't know how much it'll need, so want to make one big volume that can be used)
2. I add the name and requested storage (200 in this case)
3. In the customize tab I also check "Many Nodes Read-Write" since I'll be using the volume with multiple nodes and services I figured I needed that as well.
4. I press create and it starts with "Pending"
5. Going into the volume and opening the "Events" tab I just see "Waiting for a volume to be created, either by external provisioner "csi.hetzner.cloud" or manually created by system admininistrator"
6. I made another one for testing but this time chose to make it while making the workload, then I got this ProvisioningFailed | failed to provision volume with StorageClass "hcloud-volumes": rpc error: code = InvalidArgument desc = capability at index 1 is not supported

I'm trying to install Vaultwarden, and I first thought there was an issue with the container or workload setup that I did, but seeing that the volumes wasn't created it has to be that. Specially since I tested with rancher/hello-world. First without volume and it went perfectly. Then I added the test volume and it failed and I got the error seen in 6.

I also got shown this for Vaultwarden Running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition.

I thought it would make the volume automatically on Hetzner when I requested to make one, unless I missunderstood this

[vitobotta]

Which tool are you using? You need to create persistent volume claims to use the Hetzner storage class hcloud-volumes and the CSI driver will create the actual block storage volumes.

For example, the deployment I showed in the previous comment would become something like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hello-world
spec:
  storageClassName: hcloud-volumes
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      volumes:
      - name: hello-world
        persistentVolumeClaim:
          claimName: hello-world
      containers:
      - name: hello-world
        image: rancher/hello-world
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/test"
            name: hello-world

You need to create a PersistentVolumeClaim (not a persistent volume directly), and mount it in the pod like in the yaml above.

[Aethrexal]

I'm creating them in Rancher by just going to the "Volumes" tab then "Add Volume" which is the PersistentVolumeClaim. I created using

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: hello-world spec: storageClassName: hcloud-volumes accessModes:

• ReadWriteOnce resources: requests: storage: 3Gi

Which worked once I added a workload to it, it created it. So then I went to try the ones I created wihtout yaml and it failed. So then I made another using the form on rancher but without changing the access node i.e kept it on "Single Node Read-write" and that worked now as well. So maybe I just can't have single and many at the same time.

[Aethrexal]

Ok I just tested it with the "Many node Read-write" and that's what caused it to stay on "Pending" and fail to make a workload.

[vitobotta]

Normal block storage (such as Hetzner volumes) can only be used as "ReadWriteOnce", and only one pod at a time can access the volume. If you need to share a volume between multiple pods you need a "ReadWriteMany" volume, and that's not possible with normal block storage. For that you need to use something like NFS (in-cluster or external), CephFS, CIFS or something like that. Regular volumes can only be mounted in one pod. And in the 99% of the cases, such as with normal deployments like Vaultwarden, you need regular volumes.

[Aethrexal]

Oooh alright, make sense. Everything works now 😃 Thanks for the help again haha.

[vitobotta]

Np :)

[themaxx32000]

thank you for this tutorial – don't let this drown here but make an example readme (which is much more helpful than the 1094190284+ outdated Rancher docs & tutorials out there...)

[alissonduck]

hey @vitobotta! how are u?

First of all, I would like to say that hetzner-k3s is saving our company. it would be hard to pay the costs of self-managed kuberntes on Azure, AWS or Google. Thank you very much!

I'm still new to Kubernetes and would like to ask a question related to load balancers.

We use hetzner-k3s to run a production Kubernetes cluster, where we host some services in Ruby on Rails. All these services run on port 3000, which is the Rails default.

When I configure the Hetzner Load Balancer, by default it already adds ports 80 and 443 to the Load Balancer configurations.

I receive a request on the domain app.xyz.com, which is pointed to Balancer, it redirects to services/pods and our users automatically access our system.

My question is: Do I also need to configure port 3000 on the Hetzner Load Balancer for it to work correctly?

Our main Ruby on Rails service runs on more than 45 pods that are hosted on 7 machines with 8vpcu+32gb, but I'm not sure if I configured the load balancer correctly, given that I didn't configure port 3000 on the load balancer.

Or do just ports 80 and 443 correctly configured already route traffic between all 7 servers?

Sorry for the stupid question. I'm still at the beginning of my DevOps career haha.

[vitobotta]

hey @vitobotta! how are u?

First of all, I would like to say that hetzner-k3s is saving our company. it would be hard to pay the costs of self-managed kuberntes on Azure, AWS or Google. Thank you very much!

I'm still new to Kubernetes and would like to ask a question related to load balancers.

We use hetzner-k3s to run a production Kubernetes cluster, where we host some services in Ruby on Rails. All these services run on port 3000, which is the Rails default.

When I configure the Hetzner Load Balancer, by default it already adds ports 80 and 443 to the Load Balancer configurations.

I receive a request on the domain app.xyz.com, which is pointed to Balancer, it redirects to services/pods and our users automatically access our system.

My question is: Do I also need to configure port 3000 on the Hetzner Load Balancer for it to work correctly?

Our main Ruby on Rails service runs on more than 45 pods that are hosted on 7 machines with 8vpcu+32gb, but I'm not sure if I configured the load balancer correctly, given that I didn't configure port 3000 on the load balancer.

Or do just ports 80 and 443 correctly configured already route traffic between all 7 servers?

Sorry for the stupid question. I'm still at the beginning of my DevOps career haha.

Hi! Glad to hear it's useful :)

No, you don't need to open port 3000, that's the internal port. Did you configure the load balancer directly for the app or via an ingress controller like nginx?

The load balancer talks to the kubernetes service on nodes that depend on the configuration, and then the service is what load balances requests internally to the pods.