traefik / traefik-helm-chart

Traefik Proxy Helm Chart
https://traefik.io
Apache License 2.0
1.06k stars 747 forks source link

reusePort not working in k8s #1129

Closed Daniel-lu-zeronetworks closed 1 day ago

Daniel-lu-zeronetworks commented 2 months ago

Welcome!

What version of the Traefik's Helm Chart are you using?

traefik-29.0.0

What version of Traefik are you using?

3.0.4

What did you do?

im trying to use the same port of two different entrypoints so that I can control idletimout on different grpc streams.

i tried

          - "--entryPoints.websecure.address=:8443"
          - "--entryPoints.websecure.reusePort=true"
          - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=0"
          - "--entryPoints.websecure.transport.respondingTimeouts.writeTimeout=0"
          - "--entryPoints.websecure.transport.respondingTimeouts.idleTimeout=60"
          - "--entryPoints.websecure.lifeCycle.requestAcceptGraceTimeout=0"
          - "--entryPoints.websecure.lifeCycle.graceTimeOut=0"
          - "--entryPoints.websecure.tls=true"
          - "--entryPoints.websecure.tls.options="
          - "--entryPoints.websecure.tls.certResolver="
          - "--entryPoints.websecure.tls.domains=[]"
          - "--entryPoints.websecure.http2.maxConcurrentStreams=100"
          - "--entryPoints.banana.address=:8443"
          - "--entryPoints.banana.reusePort=true"
          - "--entryPoints.banana.transport.respondingTimeouts.readTimeout=0"
          - "--entryPoints.banana.transport.respondingTimeouts.writeTimeout=0"
          - "--entryPoints.banana.transport.respondingTimeouts.idleTimeout=60"
          - "--entryPoints.banana.lifeCycle.requestAcceptGraceTimeout=0"
          - "--entryPoints.banana.lifeCycle.graceTimeOut=0"
          - "--entryPoints.banana.tls=true"
          - "--entryPoints.banana.tls.options="
          - "--entryPoints.banana.tls.certResolver="
          - "--entryPoints.banana.tls.domains=[]"
          - "--entryPoints.banana.http2.maxConcurrentStreams=100"

What did you see instead?

i see that the streams from the banana entrypoint is still closing the stream

What is your environment & configuration?

i use k8s and deploy it via helm chart to the cluster.

Additional Information

also, it might be nice to have an explanation or documentation that states the when installed via helm entrypoints are controlled by the ports section in the values.yaml

I want to use same 443 port on two different entrypoints so that I can control the idle timeout of grpcstreams I need a few ingressroute to have a higher idletimout

also btw, when installing via helm and trying to install entrypoints with the same port even when you use resurePort in the values yaml it fails with this error :

Error: UPGRADE FAILED: failed to create resource: Service "traefik" is invalid: [spec.ports[2].nodePort: Duplicate value: 30449, spec.ports[2]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:443, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}]

this is how it looks in helm :

websecure:
    port: 8443
    reusePort: true
    expose:
      default: true
    exposedPort: 443
    protocol: TCP
    transport:
      respondingTimeouts:
        readTimeout: 0
        writeTimeout: 0
        idleTimeout: "60"
    lifeCycle:
      requestAcceptGraceTimeout: 0
      graceTimeOut: 0
    keepAliveMaxRequests: 0
    keepAliveMaxTime: 0
    tls:
      enabled: true
      options: ""
      certResolver: ""
      domains: []
    middlewares: []
    http2:
      maxConcurrentStreams: 100

  banana:
    port: 8443
    reusePort: true
    expose:
      default: true
    exposedPort: 443
    protocol: TCP
    transport:
      respondingTimeouts:
        readTimeout: 0
        writeTimeout: 0
        idleTimeout: "60"
    lifeCycle:
      requestAcceptGraceTimeout: 0
      graceTimeOut: 0
    keepAliveMaxRequests: 0
    keepAliveMaxTime: 0
    tls:
      enabled: true
      options: ""
      certResolver: ""
      domains: []
    middlewares: []
    http2:
      maxConcurrentStreams: 100
darkweaver87 commented 1 month ago

Hello @Daniel-lu-zeronetworks,

Thanks for reaching out.

reusePort is not implemented in the current version of the helm chart. You can nevertheless enable it through additionalArguments which offers users a way to set some custom traefik parameters.

The error you're facing is here because traefik handle re-using the same port whereas a kubernetes service doesn't. And actually we don't care at kubernetes service level.

Basically remove the expose section of your banana service and it will work.

Here is a tiny example with whoami.

Create a cluster:

k3d cluster create traefik --port 80:80@loadbalancer --port 443:443@loadbalancer --port 9000:9000@loadbalancer --k3s-arg "--disable=traefik@server:0"

traefik-values.yaml:

image:
  tag: v3.1.0

ingressRoute:
  dashboard:
    enabled: true

ports:
  banana:
    port: 8000
  traefik:
    port: 9000
    expose:
      default: true

additionalArguments:
  - "--entryPoints.web.reusePort=true"
  - "--entryPoints.banana.reusePort=true"

Install traefik:

helm upgrade --install --create-namespace --namespace traefik traefik traefik/traefik -f traefik-values.yaml

example.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: traefik
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: traefik
spec:
  selector:
    app: whoami
  ports:
    - protocol: TCP
      port: 80

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-web
  namespace: traefik
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`whoami-web.docker.localhost`)
      services:
        - name: whoami
          port: 80
          namespace: traefik

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-banana
  namespace: traefik
spec:
  entryPoints:
    - banana
  routes:
    - kind: Rule
      match: Host(`whoami-banana.docker.localhost`)
      services:
        - name: whoami
          port: 80
          namespace: traefik

Apply it:

kubectl apply -f example.yaml

Test it:

$ curl http://whoami-web.docker.localhost/
Hostname: whoami-6f57d5d6b5-r2hb7
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.8
IP: fe80::30a4:16ff:fea2:b10f
RemoteAddr: 10.42.0.9:46540
GET / HTTP/1.1
Host: whoami-web.docker.localhost
User-Agent: curl/8.5.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.42.0.1
X-Forwarded-Host: whoami-web.docker.localhost
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-7b9fcf4dc4-8khcc
X-Real-Ip: 10.42.0.1

$ curl http://whoami-banana.docker.localhost/
Hostname: whoami-6f57d5d6b5-75h45
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.7
IP: fe80::680d:b6ff:fe0c:34ff
RemoteAddr: 10.42.0.9:40956
GET / HTTP/1.1
Host: whoami-banana.docker.localhost
User-Agent: curl/8.5.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.42.0.1
X-Forwarded-Host: whoami-banana.docker.localhost
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-7b9fcf4dc4-8khcc
X-Real-Ip: 10.42.0.1

Technically the SO_REUSEPORT (implemented in 3.9) will make the kernel load-balance incoming TCP/UDP packets among all listening sockets. This means in the example I provided you'll have 404 responses from time to time because you'll try to make a web query on a banana port and vice-versa. To avoid this you can change the match rule by:

match: HostRegexp(`whoami-(web|banana).docker.localhost`)

Rémi