little help needed

[1mfaasj]

sorry, Ive been very busy recently, but Ive got time now 👍 I still have the problem that I can't reach the hello-world on my own domain, this works well with portforward. I doubt very much whether it is because of the way the cluster is made (firewall, loadbalancers etc) or whether it is my own fault. it would mean a lot to me if you could help me!

How I created the cluster:

---
hetzner_token: myapikeyisremoved
cluster_name: kluster
kubeconfig_path: "/cluster/kubeconfig"
k3s_version: v1.22.3+k3s1
public_ssh_key_path: "~/.ssh/id_rsa.pub"
private_ssh_key_path: "~/.ssh/id_rsa"
ssh_allowed_networks:
  - 0.0.0.0/0
verify_host_key: false
location: nbg1
schedule_workloads_on_masters: false
masters:
  instance_type: cpx21
  instance_count: 3
worker_node_pools:
- name: small
  instance_type: cpx21
  instance_count: 2

I executed the following commands to deploy nginx:

1. helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
2. helm repo update
3. helm upgrade --install --namespace ingress-nginx --create-namespace -f C:\kluster\ingress-nginx.yaml ingress-nginx ingress-nginx/ingress-nginx

How the ingress-nginx.yaml look like:

controller:
  kind: DaemonSet
  service:
    annotations:
      load-balancer.hetzner.cloud/location: nbg1
      load-balancer.hetzner.cloud/name: kluster-ingress-nginx
      load-balancer.hetzner.cloud/use-private-ip: "true"

So Iam trying to deploy this hello-world app, it is basically this file: https://gist.githubusercontent.com/vitobotta/6e73f724c5b94355ec21b9eee6f626f1/raw/3036d4c4283a08ab82b99fffea8df3dded1d1f78/deployment.yaml One thing changed though:

spec:
  rules:
  - host: mydomain.com

Everything is running well, when I Portforward the pod is gives me the Hello-world page. But I cant get this to work on my own domain...

When I descibe the hello-world it shows me 10.43.229.204, I think an internal IP right?

PS C:\kluster> kubectl describe service hello-world -n ingress-nginx
Name:              hello-world
Namespace:         ingress-nginx
Labels:            <none>
Annotations:       <none>
Selector:          app=hello-world
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.229.204
IPs:               10.43.229.204
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.4.7:80
Session Affinity:  None
Events:            <none>

When I describe the ingress used by Hello-world it gives me Host mydomain.com, so this seems to be OK right?

PS C:\kluster> kubectl describe ingress hello-world -n ingress-nginx
Name:             hello-world
Namespace:        ingress-nginx
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  mydomain.com  
              /   hello-world:80 (10.244.4.7:80)
Annotations:  <none>
Events:       <none>

When I execute this one, it gives me the IP of the external IP of the loadbalancer (162.55.152.65) This is the IP that I need to use in the DNS settings right?

PS C:\kluster> kubectl get services -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP                                   PORT(S)      
                AGE
hello-world                          ClusterIP      10.43.229.204   <none>                                        80/TCP       
                15m
ingress-nginx-controller             LoadBalancer   10.43.131.110   10.0.0.8,162.55.152.65,2a01:4f8:1c1d:201::1   80:30584/TCP,
443:31524/TCP   22m
ingress-nginx-controller-admission   ClusterIP      10.43.59.144    <none>                                        443/TCP      
                22m

How my DNS settings look like, so basically the same IP of the public loadbalancer (ingress-nginx-controller)

A | mydomain.com | 162.55.152.65
A | www | 162.55.152.65

But when I go to mydomain.com it gives me an error, nothing is there to see... Is the IP correct that Iam trying to use at the DNS settings? or do I need to add some annotations? thanks in advance

Originally posted by @jboesh in https://github.com/vitobotta/hetzner-k3s/issues/30#issuecomment-974893076

[vitobotta]

I think you need to add an annotation to the ingress resource to specify that you want to use the Nginx ingress as ingress controller. This was changed recently. So change your ingress resource like this:

...
metadata:
  name: hello-world
  annotations:
    "kubernetes.io/ingress.class": "nginx"
...

This try and let me know.

[1mfaasj]

Hi! Thanks for the reply. Ive added the annotations lines to the ingress resource, but unfortunately it didn't change anything, i still get an 400 bad request when I go to "hello-world.162.55.152.65.nip.io" in this case. (I change this to the nip.io instead my own domain for troubleshooting purposes)

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  namespace: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: ingress-nginx
spec:
  rules:
  - host: hello-world.162.55.152.65.nip.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80

When I describe the hello-world service, it gives me:

PS C:\kluster> kubectl describe service hello-world -n ingress-nginx
Name:              hello-world
Namespace:         ingress-nginx
Labels:            <none>
Annotations:       <none>
Selector:          app=hello-world
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.229.204
IPs:               10.43.229.204
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.4.7:80
Session Affinity:  None
Events:            <none>

When I describe the ingress of hello-world it gives me the correct host:

PS C:\kluster> kubectl describe ingress hello-world -n ingress-nginx
Name:             hello-world
Namespace:        ingress-nginx
Address:          10.0.0.8,162.55.152.65,2a01:4f8:1c1d:201::1
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                              Path  Backends
  ----                              ----  --------
  hello-world.162.55.152.65.nip.io  
                                    /   hello-world:80 (10.244.4.7:80)
Annotations:                        kubernetes.io/ingress.class: ingress-nginx
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    3m41s (x3 over 4m56s)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m41s (x3 over 4m56s)  nginx-ingress-controller  Scheduled for sync

the pod hello-world and ingress are running both:

PS C:\kluster> kubectl get pods -n ingress-nginx
NAME                             READY   STATUS    RESTARTS   AGE
hello-world-678c699476-zlmhl     1/1     Running   0          24h
ingress-nginx-controller-m4rd4   1/1     Running   0          24h
ingress-nginx-controller-qsvpc   1/1     Running   0          24h

When I portforward it shows me this, so the pod is healthy..

hmm what am I missing here, I don't get it...

[vitobotta]

Did you enable proxy protocol with the load balancer annotations? If yea you also need to enable it in the Mainz configmap.

[1mfaasj]

Hm I've added 3 hetzner Annotations, the cloud/location, cloud/name and use-private-ip I don't think you mean this?

PS C:\kluster> kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/version=1.0.5
                          helm.sh/chart=ingress-nginx-4.0.9
Annotations:              load-balancer.hetzner.cloud/location: nbg1
                          load-balancer.hetzner.cloud/name: kluster-ingress-nginx
                          load-balancer.hetzner.cloud/use-private-ip: true
                          meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: ingress-nginx
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.
kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.131.110
IPs:                      10.43.131.110
LoadBalancer Ingress:     162.55.152.65, 2a01:4f8:1c1d:201::1, 10.0.0.8
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30584/TCP
Endpoints:                10.244.2.4:80,10.244.4.5:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31524/TCP
Endpoints:                10.244.2.4:443,10.244.4.5:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

[vitobotta]

Can you share the full manifests of the various resources you've created?

[vitobotta]

When you visit the IP directly, not the domain, what do you see?

[1mfaasj]

the error is the same as going to "hello-world.162.55.152.65.nip.io" or "162.55.152.65"

400 Bad Request
nginx

This is my ingress-nginx.yaml file: https://file.re/2021/11/22/ingress-nginx/

and the hello-world yaml (including the deployment, service and ingress): https://file.re/2021/11/22/hello-world/

So what I did are these steps (nothing more):

1. helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
2. helm repo update
3. helm upgrade --install --namespace ingress-nginx --create-namespace -f C:\kluster\ingress-nginx.yaml ingress-nginx ingress-nginx/ingress-nginx
4. kubectl apply -f hello-world.yaml

[vitobotta]

Can you check the nginx logs? 400 status code happens when for example the proxy protocol is enabled on the load balancer but not in the nginx configmap, or when you are making https requests to an http backend or the other way around.

I am wondering if for some reason the proxy protocol is enabled on the load balancer.

Can you share the output of kubectl -n ingress-nginx get cm ingress-nginx-controller -oyaml?

You can try adding the setting use-proxy-protocol: "true" to that config map to see if the problem is related to the proxy protocol as I am suspecting. If you update the config map with that setting the nginx config will be automatically reloaded and you can try accessing the ingress resource again from the domain.

[1mfaasj]

This is the output:

PS C:\kluster> kubectl -n ingress-nginx get cm ingress-nginx-controller -oyaml
apiVersion: v1
data:
  allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
  creationTimestamp: "2021-11-21T19:59:16Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 1.0.5
    helm.sh/chart: ingress-nginx-4.0.9
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "1694"
  uid: 144f0834-f352-47f1-b167-74b644e9d2be

yes I will try to add: load-balancer.hetzner.cloud/uses-proxyprotocol: 'true' to the ingress-nginx.yaml.

[vitobotta]

Make sure you edit the config map also, like I explained in the previous comment. Also check in the Hetzner Console if the proxy protocol is enabled on the load balancer, so to confirm.

[1mfaasj]

i think i need a little explaination to edit the config map, so it is not the 'uses-proxyprotocol' line? This is what I have now:

controller:
  kind: DaemonSet
  service:
    annotations:
      load-balancer.hetzner.cloud/location: nbg1
      load-balancer.hetzner.cloud/name: kluster-ingress-nginx
      load-balancer.hetzner.cloud/use-private-ip: "true"
      load-balancer.hetzner.cloud/uses-proxyprotocol: 'true'

In the console, I cant find the proxy protocol Edit: When I click on Service on the Loadbalancer, it shows me: enabled.

[vitobotta]

Ok, the proxy protocol is enabled. Now run kubectl -n ingress-nginx edit cm ingress-nginx-controller to edit the config map I mentioned, and add use-proxy-protocol: "true" to the data section. BTW these issues are not related to the tool :)

[1mfaasj]

OK it is working! you've just made my day. thanks for your time. for my own learning; is the proxy protocol only needed for exposing the real IP address of the client in applications that I deploy to the cluster?

[vitobotta]

Nice! Glad you got it working. Yes, the proxy protocol is required to ensure you always get the actual IP of the user and not the IP of a load balancer in between (either the load balancer for the ingress or some internal loadbalacing in kubernetes).

If you don't care about that you can disable the proxy protocol but be sure you do so both in the load balancer annotations and in the config map.

[1mfaasj]

thanks for the explaination! in the meanwhile Iam trying to deploy an oauth service in the cluster to be able to add some authentication later to the apps. but when I search the internet for examples, it's still complex for me May I ask what kind of authentication you are using (for private apps)? Do you mind sharing a good example of the deployment?

I notice that it is very difficult to find good documentation about this in general... I'm coming from Docker compose and swarm and there's plenty of documentation about that! I'm a little disappointed with Kubernetes.

[vitobotta]

What kid of applications?

[1mfaasj]

What kid of applications?

Like jackett (torrent tracker) for instance or homer (dashboard), they dont have their own authentication so i need something to make it private and protected.

[vitobotta]

You can enable http basic authentication with the ingress resource. See https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

[1mfaasj]

hm when I try to execute this command: htpasswd -c auth foo the result is:

htpasswd : The term 'htpasswd' is not recognized as the name of a cmdlet, function, script file, or operable program.
 Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

I cannot use htpasswd in the cluster I guess? I'm using Windows 11 which I connect to the cluster, and htpasswd is a Linux thing right? maybe this has something to do with it. but the VMs in hetnzer are ubuntu right?

[vitobotta]

htpasswd is only needed to generate the encoded string. If it's not available on Windows you can try some other utility, e.g. a quick googling showed this at the top https://htpasswdgenerator.com/download_htpasswd_generator.html

[1mfaasj]

ah nice it is working! this is basic authentication right, it would be nicer to have something in place to use oauth2-proxy i was hoping you had an example for deployment and ingress for an test application

Iam trying for days now to get oauth2-proxy working and created a ticket for help https://github.com/kubernetes/ingress-nginx/issues/7987 I just don't think I'll get a response anytime soon.

[vitobotta]

At ingress level you can only do basic http authentication. Anything else must be implemented at application level I'm afraid. At least AFAIK.

[1mfaasj]

As far as I know the oauth2-proxy consists of a deployment, service and ingress. if you have a particular application that you want to add authentication to, you need to add annotations to that ingress.

example ( but not 100% working yet ): https://github.com/jboesh/test

[vitobotta]

Cool, I haven't come across that before :)

[1mfaasj]

basically this page. but its not working yet in my case https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/

[1mfaasj]

this has nothing to do with the creating of the cluster,I was just wondering if you had something in place already so you can close the ticket if you want, thanks for all the help!

[vitobotta]

No problem. I'll close this one but let me know if you run into any issues. Cheers