search

techno-tim / k3s-ansible

The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more. Build. Destroy. Repeat.
https://technotim.live/posts/k3s-etcd-ansible/
Apache License 2.0
2.41k stars 1.05k forks source link

X-Real-Ip from traefik reverse proxy is wrong #281

Closed madmurl0c closed 1 year ago

madmurl0c commented 1 year ago

Hey everyone, I'm new to k8s and tried setting it up in my homelab today. Thanks to this repository it was quite easy to get started but I'm hitting a roadblock that I cannot seem to resolve on my own. I made a small example deployment shown below but I cannot get the clients ip address from the X-Real-Ip header.

Is there any additional configuration I forgot to add? If so, would it be useful to incorporate it into this repository?

Thanks for your help :)

Expected Behavior

X-Real-Ip should contain the client IP

Current Behavior

X-Real-Ip contains an ip from the k8s range (10.42.2.0)

Steps to Reproduce

  1. Run kubectly apply -f example-deployment.yaml
  2. Open nginx.my.lan in your browser

example-deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: containous/whoami
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: nginx
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik-external
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`nginx.my.lan`)
      kind: Rule
      services:
        - name: nginx
          port: 80
      middlewares:
        - name: default-headers

Context (variables)

Operating system: Ubuntu 22.04 LTS

Hardware: VM

Variables Used

all.yml

k3s_version: "v1.24.12+k3s1"
ansible_user: NA
systemd_dir: "/etc/systemd/system"

flannel_iface: "ens18"

apiserver_endpoint: "192.168.3.1"

k3s_token: "NA"

extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
  --kube-apiserver-arg default-not-ready-toleration-seconds=60
  --kube-apiserver-arg default-unreachable-toleration-seconds=60
  --kube-controller-arg node-monitor-period=40s
  --kube-controller-arg node-monitor-grace-period=40s
  --kubelet-arg node-status-update-frequency=10s
extra_agent_args: >-
  {{ extra_args }}
  --kubelet-arg node-status-update-frequency=10s

kube_vip_tag_version: "v0.5.11"

metal_lb_speaker_tag_version: "v0.13.9"
metal_lb_controller_tag_version: "v0.13.9"

metal_lb_ip_range: "192.168.3.2-192.168.3.254"

Hosts

host.ini

[master]
192.168.2.11
192.168.2.12
192.168.2.13

[node]
192.168.2.14
192.168.2.15
192.168.2.16

[k3s_cluster:children]
master
node

Possible Solution

According to this (https://stackoverflow.com/questions/50585616/kubernetes-metallb-traefik-how-to-get-real-client-ip) I think metal_lb_mode: "layer2" must be switched to metal_lb_mode: "bgp" but I'm not sure what side effects this could/would have.