502 Gateway Not Found - Simple HTTP App, Authentik, Outpost, K8s, Nginx Ingress Controller

[ServerNinja]

Describe your question/ I'm trying to configure an authentik outpost for a single simple http app (no built in auth) that needs to use the single application forward auth provider on a k8s cluster using the ingress-nginx controller. This is actually the "alertmanager" and "prometheus" console apps, which have no built-in authentication mechanisms.

I've been struggling with setting this up for days. Right now, I do get a gateway 502 error when the application redirects to the outpost. When I curl the /outpost.goauthentik.io/ping url on the app, it gives me the http 204 that is expected in the troubleshooting section in the docs for this.

Version and Deployment:

• authentik version: 2024.83
• Deployment: helm on AWS EKS (AWS Kubernetes Flavor)
• Kubernetes v1.30

Relevant Info

• I am creating an outpost via the authentik console and not using the embedded outpost. However, I DID NOT delete the embedded outpost from the configuration (basically leaving it alone)
• Using the ingress-nginx controller for ingress. Ingress configured to use AWS load balancers with LetsEncrypt certs via cert-manager

Outposts console (The outpost in question is the one named: "alertmanager"):

Outpost Configuration:

log_level: trace
authentik_host: https://authentik.xxxxxxxxxx.com
refresh_interval: minutes=5
kubernetes_replicas: 1
kubernetes_namespace: authentik
authentik_host_browser: https://alertmanager.xxxxxxxxxx.com
object_naming_template: ak-outpost-%(name)s
authentik_host_insecure: true
kubernetes_service_type: ClusterIP
kubernetes_disabled_components:
  - ingress
  - traefik middleware

Alertmanager Provier and Application:

Ingress for Alertmanager

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-url: "http://ak-outpost-alertmanager.authentik.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx"
    nginx.ingress.kubernetes.io/auth-signin: "/outpost.goauthentik.io/start?rd=$escaped_request_uri"
    nginx.ingress.kubernetes.io/auth-response-headers: Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
    nginx.ingress.kubernetes.io/auth-snippet: |
      proxy_set_header X-Forwarded-Host $http_host;
#    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  name: alertmanager-ingress
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
  - host: alertmanager.xxxxxxxxxx.com
    http:
      paths:
      - backend:
          service:
            name: kube-prometheus-stack-alertmanager
            port:
              number: 9093
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - alertmanager.xxxxxxxxxx.com
    secretName: wildcard-certificate

Ingress for the Alertmanager Outpost:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  name: alertmanager-outpost-ingress
  namespace: authentik
spec:
  ingressClassName: nginx
  rules:
  - host: alertmanager.xxxxxxxxxx.com
    http:
      paths:
      - backend:
          service:
            name: ak-outpost-alertmanager
            port:
              number: 9000
        path: /outpost.goauthentik.io
        pathType: Prefix
  tls:
  - hosts:
    - alertmanager.xxxxxxxxxx.com
    secretName: wildcard-certificate

**Screenshot and Trace Logs when trying to access https://alertmanager.xxxxxxxxxx.com:"

The Trace Log: authentik_log.txt

[ServerNinja]

Update:

Added ingress annotations to help address the following error:

upstream sent too big header while reading response header from upstream

Annotations:

    nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"

Now I'm getting lots of these errors:

E1015 13:54:52.885570       7 tcp.go:129] Error copying data: readfrom tcp 127.0.0.1:40012->127.0.0.1:442: use of closed network connection

[ServerNinja]

This seems to be doing a lot of 302 redirects, looping between autorize/ and start/. Is this a config issue with the ingress for alertmanager and the outpost?