Proxy outpost doesn't seem to be handling request correctly

[justSem]

Describe the bug Let me start off by saying a few things:

• I realize the title is rather vague, but I'm not sure how to describe it.
• I'm not sure if this is an issue with authentik, or nginx, the logging is rather inconclusive to me, so I'm asking help here.

Since the latest authentik patch I've suddenly ran into the issue of all my forward-auth applications becoming accessible without authentication. I'm not seeing any errors, or timeouts, it just seems to have stopped working out of the blue.

So, right now I'm asking for help to see how I can approach getting to the bottom of this. So far I've looked at both nginx and authentik logging. Both seem fine.

When accessing a protected application, I see log entries appearing in the outpost logs, regarding that vhost, indicating to me that nginx is passing traffic to the outpost. So far I've tried:

• Completely reinstalling authentik
• Reinstalling my protected applications
• Recreating my ingresses.

Expected behavior I expect a login page for authentik instead of being shown the application. (Yes, I've tested with other browsers, IPs and systems)

Ingress configuraiton

Application:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: le-justsem
    nginx.ingress.kubernetes.io/auth-response-headers: Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
    nginx.ingress.kubernetes.io/auth-signin: https://<redacted>.justsem.org/outpost.goauthentik.io/start?rd=$escaped_request_uri
    nginx.ingress.kubernetes.io/auth-snippet: |
      proxy_set_header X-Forwarded-Host $http_host;
    nginx.ingress.kubernetes.io/auth-url: http://ak-outpost-default.security.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/cors-expose-headers: '*'
  labels:
    app.kubernetes.io/instance: <redacted>
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: <redacted>
    helm.sh/chart: app-template-3.4.0
    helm.toolkit.fluxcd.io/name: <redacted>
    helm.toolkit.fluxcd.io/namespace: <redacted>
  name: <redacted>-app
  namespace: <redacted>
spec:
  ingressClassName: external-nginx
  rules:
  - host: <redacted>.justsem.org
    http:
      paths:
      - backend:
          service:
            name: <redacted>
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - <redacted>.justsem.org
    secretName: <redacted>-cert

Outpost:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    traefik.ingress.kubernetes.io/affinity: "true"
    traefik.ingress.kubernetes.io/router.tls: "true"
  labels:
    app.kubernetes.io/instance: default
    app.kubernetes.io/managed-by: goauthentik.io
    app.kubernetes.io/name: authentik-proxy
    app.kubernetes.io/version: 2024.8.3
    goauthentik.io/outpost-name: default
    goauthentik.io/outpost-type: proxy
    goauthentik.io/outpost-uuid: 15c6346be9084b30ac2a6fbefe8bd261
  name: ak-outpost-default
  namespace: security

spec:
  ingressClassName: external-nginx
  rules:
  - host: <redacted>.justsem.org
    http:
      paths:
      - backend:
          service:
            name: ak-outpost-default
            port:
              name: http
        path: /outpost.goauthentik.io
        pathType: Prefix

  tls:
  - hosts:
    - <redacted>.justsem.org
    ......
    secretName: authentik-cert

Logs Outpost:

{"event":"passing to application mux","host":"<redacted>.justsem.org","level":"trace","logger":"authentik.outpost.proxyv2","timestamp":"2024-10-06T16:10:20+02:00"}
{"event":"tracing headers for debug","header":{"Accept":["*/*"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-GB"],"Connection":["close"],"Priority":["u=4, i"],"Referer":["https://<redacted>.justsem.org/"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"X-Api-Key":["2b64477f876d42d98465bd689dd0d5c9"],"X-Auth-Request-Redirect":["/api/v3/command"],"X-Forwarded-For":["<redacted>"],"X-Forwarded-Host":["<redacted>.justsem.org"],"X-Original-Method":["GET"],"X-Original-Url":["https://<redacted>.justsem.org/api/v3/command"],"X-Real-Ip":["<redacted>"],"X-Request-Id":["feb04f273fa55649c3651acac19080d6"],"X-Requested-With":["XMLHttpRequest"],"X-Sent-From":["nginx-ingress-controller"]},"level":"trace","logger":"authentik.outpost.proxyv2.application","name":"sonarr","timestamp":"2024-10-06T16:10:20+02:00"}
{"event":"nginx forwarded url","level":"trace","logger":"authentik.outpost.proxyv2.application","name":"sonarr","timestamp":"2024-10-06T16:10:20+02:00","url":"https://<redacted>.justsem.org/api/v3/command"}
{"event":"cookie encode","level":"trace","timestamp":"2024-10-06T16:10:20+02:00"}
{"event":"cookie encode","level":"trace","timestamp":"2024-10-06T16:10:20+02:00"}

nginx:

{"time": "2024-10-06T16:13:22+02:00", "remote_addr": "", "x_forwarded_for": "<redacted>", "request_id": "19d7298ad279a6847de36675b2bd53c9", "remote_user": "", "bytes_sent": 1186, "request_time": 0.092, "status": 200, "vhost": "<redacted>.justsem.org", "request_proto": "HTTP/2.0", "path": "/api/v3/command", "request_query": "", "request_length": 30, "duration": 0.092,"method": "GET", "http_referrer": "https://<redacted>.justsem.org/", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"}

Version and Deployment (please complete the following information):

• authentik version: 2024.8.3
• Deployment: Helm
• Kubernetes: v1.31.1
• CNI: Cilium v1.16.1

[justSem]

Eventually this turned out to be an issue with the ingress-nginx's default backend returning everything as HTTP 200; instead of the appropriate error code; thus making nginx think the authentication had succeeded.