Closed Aethrexal closed 3 years ago
Not sure I understand your questions 100% but let's try. So there's two things for which you need a load balancer. First, for the Kubernetes API server only if you create a cluster with multiple masters; in this case, you don't have to do anything as the installer automatically creates and configures a load balancer for the API server.
Then there are load balancers for your workloads. The installer installs the cloud controller manager that allows you to create Kubernetes services of type load balancer out of the box, but you need to specify at least two annotations: 1) one annotation which sets the Hezner location, otherwise the load balancer won't be provisioned and will stay in "pending" state.; 2) the annotation that ensures that the communication between the load balancer and the cluster nodes happens through the private network. This is to avoid opening additional ports on the nodes. So at a minimum when you create a service of type LoadBalancer you need these annotations:
service:
annotations:
load-balancer.hetzner.cloud/location: nbg1
load-balancer.hetzner.cloud/use-private-ip: "true"
The other annotations I mentioned in the README are only needed if you want your apps to "see" the real IP of the client, because otherwise they will see the IP of the load balancer. But I suggest you skip this for now until you are more familiar with this.
Since I only deal with regular web apps, what I usually do in my clusters is install the Nginx ingress controller and let Kubernetes create a load balancer for it. Then for each web app I create an ingress resource, so all of them will share a single load balancer in the end, which is cheaper and easier.
To easily install the Nginx ingress controller you can do this. First, run the following to create a yaml file with the values for the Nginx ingress Helm chart:
cat << EOF > /tmp/ingress-nginx.yaml
controller:
kind: DaemonSet
service:
annotations:
load-balancer.hetzner.cloud/location: nbg1
load-balancer.hetzner.cloud/name: cluster-name-ingress-nginx
load-balancer.hetzner.cloud/use-private-ip: "true"
EOF
And then install the Nginx ingress controller using that configuration:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install \
--namespace ingress-nginx \
--create-namespace \
-f /tmp/ingress-nginx.yaml \
ingress-nginx ingress-nginx/ingress-nginx
Since you've specified at least the Hetzner location, the cloud controller manager will correctly create a load balancer for Nginx. It usually takes a minute or so. You can check from the Hetzer console but it takes a 1-2 minutes for the load balancer to get the IP as seen by Kubernetes. You can check with kubectl:
kubectl -n ingress-nginx get svc
After one or two minutes the LoadBalancer service should no longer be "pending" and should have a public IP assigned to it. That means that you are ready to create ingress resources. So you can use that public IP to configure DNS records for your apps.
Now, look at this example deployment: https://gist.githubusercontent.com/vitobotta/6e73f724c5b94355ec21b9eee6f626f1/raw/3036d4c4283a08ab82b99fffea8df3dded1d1f78/deployment.yaml
As you can see there's a service for the deployment, and an ingress resource. The service is of type ClusterIP by default because we don't need a load balancer for it, since it will be sharing the load balancer created for the ingress controller (nginx) with other services.
Look at the ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-world
spec:
rules:
- host: hello-world.<load balancer ip>.nip.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hello-world
port:
number: 80
Once you replace <load balancer ip>
with the public IP assigned to the load balancer created automatically for Nginx, your sample deployment will be reachable at the url http://hello-world.load balancer-ip.nip.io - nip.io, if you have never heard of it, is just a quick way to test things without configuring DNS (a query to a hostname ending in nip.io simply returns the IP address it finds in the hostname itself).
So as you can see you only need to create a load balancer when you install the ingress controller. Nginx, and that load balancer is created automatically for you. So you really don't need to do anything else / you don't need to deal with load balancers anymore. Just create one ingress resource for each web app and you're done.
Now, if you want provision TLS certificates with cert manager, you first install the helm chart:
helm repo add jetstack https://charts.jetstack.io
helm upgrade --install \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true \
cert-manager jetstack/cert-manager
Then you need to configure one or more issuers or cluster issuers which will take care of doing stuff to obtain certificates. For example for production certificates you would run this:
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
email: your@email.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod-account-key
solvers:
- http01:
ingress:
class: nginx
EOF
This creates a cluster issuer that can be used in any namespace. All you need to do in order to trigger the issuance of a certificate is update the ingress resource for your app with two changes:
So the ingress resource as in the same deployment I linked to earlier would become:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-world
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/tls-acme: "true"
spec:
rules:
- host: yourdomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hello-world
port:
number: 80
tls:
- hosts:
- yourdomain.com
secretName: yourdomain-tls
Ensure that before creating or updating the ingress resource, you have already configured the DNS so that yourdomain.com points to the IP of the Nginx load balancer.
Within a couple of minutes, the certificate is provisioned and the app should be reachable at https://yourdomain.com.
These are the basics. Like I mentioned in the REAME if you want your apps to see the client's real IP address you need a few more annotations on the Nginx service, which I explained in the README.
As for Rancher, once you have the ingress controller (Nginx) and cert-manager up and running, it's easy to install with Helm:
helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
helm repo update
kubectl create namespace cattle-system
helm upgrade --install \
--namespace cattle-system \
--set hostname=rancher.yourdomain.com \
--set ingress.tls.source=letsEncrypt \
--set letsEncrypt.email=your@email.com \
rancher \
rancher-stable/rancher
With the commands above, and specifying your actual Rancher domain and your email address (for cert-manager notifications), Rancher should be up and running in a few minutes. Just ensure you configure the DNS for the Rancher domain in advance so that it points to the Nginx load balancer IP.
Wait until the deployment is ready (kubectl -n cattle-system rollout status deploy/rancher
) and then Rancher should be reachable at https://rancher.yourdomain.com.
Remember to create a new admin and disable the default admin.
This should help you get started hopefully :)
This is exactly what I needed! Thank you so much 😃
First, for the Kubernetes API server only if you create a cluster with multiple masters; in this case, you don't have to do anything as the installer automatically creates and configures a load balancer for the API server.
Then there are load balancers for your workloads.
This is one point where I was confused the most. Cause I thought the load balancer that got created automatically for the API was also the same that would be used for making DNS for apps. I just couldn't get it all to fit together reading all the different docs and watching different videos etc haha.
So I was just at a complete stand still, and this just made everything so much clearer for me. So hopefully it should all go much smoother for me now haha. Thanks again for the help!
Nope, the API server requires its own load balancer, that is unrelated to the load balancer used by the apps. Note that the installer only creates the LB for the API server if you specify more than 1 master, otherwise it doesn't and configures the kubeconfig to connect directly to the single master. Let me know if you have other questions otherwise I will close this. :)
Oh make sense 😃
That would be all but, I might be missing something cause it's not creating the volumes.
ProvisioningFailed | failed to provision volume with StorageClass "hcloud-volumes": rpc error: code = InvalidArgument desc = capability at index 1 is not supported
I'm trying to install Vaultwarden, and I first thought there was an issue with the container or workload setup that I did, but seeing that the volumes wasn't created it has to be that. Specially since I tested with rancher/hello-world
. First without volume and it went perfectly. Then I added the test volume and it failed and I got the error seen in 6.
I also got shown this for Vaultwarden Running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
.
I thought it would make the volume automatically on Hetzner when I requested to make one, unless I missunderstood this
Which tool are you using? You need to create persistent volume claims to use the Hetzner storage class hcloud-volumes
and the CSI driver will create the actual block storage volumes.
For example, the deployment I showed in the previous comment would become something like this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hello-world
spec:
storageClassName: hcloud-volumes
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
labels:
app: hello-world
spec:
replicas: 1
selector:
matchLabels:
app: hello-world
template:
metadata:
labels:
app: hello-world
spec:
volumes:
- name: hello-world
persistentVolumeClaim:
claimName: hello-world
containers:
- name: hello-world
image: rancher/hello-world
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/test"
name: hello-world
You need to create a PersistentVolumeClaim (not a persistent volume directly), and mount it in the pod like in the yaml above.
I'm creating them in Rancher by just going to the "Volumes" tab then "Add Volume" which is the PersistentVolumeClaim. I created using
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: hello-world spec: storageClassName: hcloud-volumes accessModes:
- ReadWriteOnce resources: requests: storage: 3Gi
Which worked once I added a workload to it, it created it. So then I went to try the ones I created wihtout yaml and it failed. So then I made another using the form on rancher but without changing the access node i.e kept it on "Single Node Read-write" and that worked now as well. So maybe I just can't have single and many at the same time.
Ok I just tested it with the "Many node Read-write" and that's what caused it to stay on "Pending" and fail to make a workload.
Normal block storage (such as Hetzner volumes) can only be used as "ReadWriteOnce", and only one pod at a time can access the volume. If you need to share a volume between multiple pods you need a "ReadWriteMany" volume, and that's not possible with normal block storage. For that you need to use something like NFS (in-cluster or external), CephFS, CIFS or something like that. Regular volumes can only be mounted in one pod. And in the 99% of the cases, such as with normal deployments like Vaultwarden, you need regular volumes.
Oooh alright, make sense. Everything works now 😃 Thanks for the help again haha.
Np :)
thank you for this tutorial – don't let this drown here but make an example readme (which is much more helpful than the 1094190284+ outdated Rancher docs & tutorials out there...)
hey @vitobotta! how are u?
First of all, I would like to say that hetzner-k3s is saving our company. it would be hard to pay the costs of self-managed kuberntes on Azure, AWS or Google. Thank you very much!
I'm still new to Kubernetes and would like to ask a question related to load balancers.
We use hetzner-k3s to run a production Kubernetes cluster, where we host some services in Ruby on Rails. All these services run on port 3000, which is the Rails default.
When I configure the Hetzner Load Balancer, by default it already adds ports 80 and 443 to the Load Balancer configurations.
I receive a request on the domain app.xyz.com, which is pointed to Balancer, it redirects to services/pods and our users automatically access our system.
My question is: Do I also need to configure port 3000 on the Hetzner Load Balancer for it to work correctly?
Our main Ruby on Rails service runs on more than 45 pods that are hosted on 7 machines with 8vpcu+32gb, but I'm not sure if I configured the load balancer correctly, given that I didn't configure port 3000 on the load balancer.
Or do just ports 80 and 443 correctly configured already route traffic between all 7 servers?
Sorry for the stupid question. I'm still at the beginning of my DevOps career haha.
hey @vitobotta! how are u?
First of all, I would like to say that hetzner-k3s is saving our company. it would be hard to pay the costs of self-managed kuberntes on Azure, AWS or Google. Thank you very much!
I'm still new to Kubernetes and would like to ask a question related to load balancers.
We use hetzner-k3s to run a production Kubernetes cluster, where we host some services in Ruby on Rails. All these services run on port 3000, which is the Rails default.
When I configure the Hetzner Load Balancer, by default it already adds ports 80 and 443 to the Load Balancer configurations.
I receive a request on the domain app.xyz.com, which is pointed to Balancer, it redirects to services/pods and our users automatically access our system.
My question is: Do I also need to configure port 3000 on the Hetzner Load Balancer for it to work correctly?
Our main Ruby on Rails service runs on more than 45 pods that are hosted on 7 machines with 8vpcu+32gb, but I'm not sure if I configured the load balancer correctly, given that I didn't configure port 3000 on the load balancer.
Or do just ports 80 and 443 correctly configured already route traffic between all 7 servers?
Sorry for the stupid question. I'm still at the beginning of my DevOps career haha.
Hi! Glad to hear it's useful :)
No, you don't need to open port 3000, that's the internal port. Did you configure the load balancer directly for the app or via an ingress controller like nginx?
The load balancer talks to the kubernetes service on nodes that depend on the configuration, and then the service is what load balances requests internally to the pods.
Edit: Made some parts easier to read.
This isn't an issue with this script or anything, it seems that I'm just very stupid 😓, I don't know where else to ask except here since using this script and easier point to go from.
So, I give up lol. I've looked through so many docs and guides and what not but, I'm so confused how I'm suppose to get rancher to work. I mean, rancher and cert-manager boots up and the pods gets ready, but the whole load balancers and that confuses the hell out of me. Load Balancers It's that part that confuses me a lot. Where I would place those and so on.
I thought I made progress when I found The load balancer README on hcloud CCM, but even that confuses me and I'm not sure how I'm suppose to set it all up going from this script. I do feel like I still made some progress, specially also using Lens as that made it a bit easier to have a look at everything going on with the cluster.
I assume the hostname would be
load.example.com
with setting up the DNS on cloudflare, with typeA
whereload.example.com
points to Load balancerPublic IP
. I might be wrong with even that.So basically, would be forever grateful with some help and guidance with this. Cause either I'm missing something in all docs and guides I've been reading, or I can't read, or I'm simply just stupid lol.