Embedded Outpost returning 404

[philnickl]

I have been unable to configure the embedded outpost to work with NGINX Forward Auth. Curl requests when pinging the outpost return a 404 (Authentik 404 page, not a NGINX 404 page). SAML/oAuth2 Applications already configured in Authentik continue to work fine.

Accessing the endpoint service returns a 500 NGINX error - due to authentik returning a 404 in the auth-url I believe.

2024/04/03 21:17:05 [error] 27#27: *3494 auth request unexpected status: 404 while sending to client, client: 10.0.1.141

The Embedded Outpost config appears to be correct

authentik_host: https://auth.tld.com/
docker_network: null
container_image: null
docker_map_ports: true
kubernetes_replicas: 1
kubernetes_namespace: default
authentik_host_browser: https://auth.tld.com/

location / {
        include /etc/nginx/proxy.conf;
        include /etc/nginx/resolver.conf;
        set $upstream_app paperless;
        set $upstream_port 8000;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        ##############################
        # authentik-specific config
        ##############################
        auth_request     /outpost.goauthentik.io/auth/nginx;
        error_page       401 = @goauthentik_proxy_signin;
        auth_request_set $auth_cookie $upstream_http_set_cookie;
        add_header       Set-Cookie $auth_cookie;

        # translate headers from the outposts back to the actual upstream
        auth_request_set $authentik_username $upstream_http_x_authentik_username;
        auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
        auth_request_set $authentik_email $upstream_http_x_authentik_email;
        auth_request_set $authentik_name $upstream_http_x_authentik_name;
        auth_request_set $authentik_uid $upstream_http_x_authentik_uid;

        proxy_set_header X-authentik-username $authentik_username;
        proxy_set_header X-authentik-groups $authentik_groups;
        proxy_set_header X-authentik-email $authentik_email;
        proxy_set_header X-authentik-name $authentik_name;
        proxy_set_header X-authentik-uid $authentik_uid;
    }

    # all requests to /outpost.goauthentik.io must be accessible without authentication
    location /outpost.goauthentik.io {
        proxy_pass              http://ubuntu-2311:9000/outpost.goauthentik.io;
        # ensure the host of this vserver matches your external URL you've configured
        # in authentik
        proxy_ssl_verify              off;
        proxy_set_header        Host $host;
        proxy_set_header        X-Original-URL $scheme://$http_host$request_uri;
        add_header              Set-Cookie $auth_cookie;
        auth_request_set        $auth_cookie $upstream_http_set_cookie;
        proxy_pass_request_body off;
        proxy_set_header        Content-Length "";
    }

    # Special location for when the /auth endpoint returns a 401,
    # redirect to the /start URL which initiates SSO
    location @goauthentik_proxy_signin {
        internal;
        add_header Set-Cookie $auth_cookie;
        return 302 /outpost.goauthentik.io/start?rd=$request_uri;
        # For domain level, use the below error_page to redirect to your authentik server with the full redirect path
        # return 302 https://authentik.company/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
    }
}

[ThauMish]

Hi, I got the exact same issue, same config.

[philnickl]

Appears the issue is already noted here #8956

[twk3]

Had the same issue an needed to swap to version 2023.10 to get it working. With trace logging turned on 2024.2.2 doesn't appear to log anything related to trying to find the app when hitting /output.authentik.io/auth/nginx it just logs the headers (showing a host that should match my app) then immediately issues the 404 without ever indicating whether it tried to do anything, no app ended up associated with the log output.

[cougz]

I have the same issue with 2024.2. It works just fine on 2023.10.7.

[BeryJu]

please provide the logs of your server container

[cougz]

When accessing the embedded outpost the only log row I see in the server container is:

{"auth_via": "unauthenticated", "domain_url": "redacted.com", "event": "/outpost.goauthentik.io/auth/nginx", "host": "redacted.com", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 1753928, "remote": "10.40.33.202", "request_id": "8a56ee9c20644100a3722f22e3899dcd", "runtime": 11, "schema_name": "public", "scheme": "http", "status": 404, "timestamp": "2024-05-16T08:18:14.785086", "user": "", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}

[malwarepad]

This is indeed a problem for me on a clean install

[HeavyTobi]

For me the following made it work again after almost eating my keyboard from frustration:

• go to Admin Area --> Applications --> Outposts --> authentik Embedded
• Actions -> Edit
• make sure your application is in the selected applications list
• 

after doing so it worked fine for me, hope this helps you.

It's been a while since I've last did stuff with the forward auth stuff but I don't think I remember that this was required earlier

[wrenix]

Same here - @HeavyTobi they are till in the "Selected Applications" -> moving around does not help

[vincentscode]

The problem appears to only occur when using single application mode instead of domain mode (which currently breaks redirects).

[matthewmax09]

Sharing that I followed the fixed Nginx configuration template for forward auth in #10181. Forward auth is working for me now. The embedded outpost no longer returns a 404.

[Schachte]

embedded outpost 404 is breaking me as well on clean install. @BeryJu any recommendations or things to know when using the embedded outpost with Docker? Been debugging for many hours

[engAmirEng]

this is not a question rather a bug, /outpost.goauthentik.io/ping is returning 404 with not even a single line of log version: 2024.6.3 @philnickl

[Schachte]

@engAmirEng I got something working this morning, maybe I can write something up to show end to end setup. The logs were a bit sparse, but instead of using the embedded outpost in core, I deployed a standalone outpost and it seems to be working. Debugging was a tad better because AUTHENTIK_DEBUG to "true" got me a bit further.

[andres-asm]

I am getting 404s too. If I delete the embedded outpost and restart my container it's recreated.

Then I get this when I try https://auth.domain/outpost.goauthentik.io

As soon as I modify the outpost to include the application I get a 404

[andres-asm]

upgraded to the latest version that doesn't require a db update, I still get this

[Schachte]

@andres-asm Are you running embedded outpost or external outpost? Have you added debug logs to nginx?

Add this to your http block.

    log_format detailed_headers_log '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr: $request $status '
                                    'Host: "$host" '
                                    'X-Real-IP: "$remote_addr" '
                                    'X-Forwarded-For: "$proxy_add_x_forwarded_for" '
                                    'X-Forwarded-Proto: "$scheme" '
                                    'X-Forwarded-Host: "$host" '
                                    'X-Forwarded-Port: "$server_port" '
                                    'X-Original-URL: "$scheme://$http_host$request_uri" '
                                    'http_host: "$http_host" '
                                    'request_uri: "$request_uri" '
                                    'cookie: "$http_cookie" '
                                    'set_cookie: "$sent_http_set_cookie" '
                                    'upstream_http_set_cookie: "$upstream_http_set_cookie"';

In each location block, you can get detailed debug logs by dropping in: access_log /var/log/nginx/authentik_access_debug.log detailed_headers_log;

You can debug these directly with tail -f /var/log/nginx/authentik_access_debug.log

[Schachte]

Ok, got this working using embedded outpost with NGINX on latest version Notes: https://ryan-schachte.com/notes/authentik_embedded_outpost_docker/

@andres-asm & @engAmirEng let me know if this helps you as I spent a few hours debugging to get this working. Got manual outpost and embedded working fine with latest version.

The documentation could use some work in this area.

[engAmirEng]

@Schachte my problem is authentik.myhost.domain/outpost.goauthentik.io/auth/nginx retuning 404, what did you exactly solved about this?

[Schachte]

@Schachte my problem is authentik.myhost.domain/outpost.goauthentik.io/auth/nginx retuning 404, what did you exactly solved about this?

Did you review the NGINX and Docker config from my link? You most likely have a misconfiguration as I was getting that previously as well. I documented the changes that resolved it for me.

[andres-asm]

@Schachte the issue is with the embedded outpost. I actually added another outpost to the same docker compose file and got everything working with that other outpost.

I tried on a default docker setup, as soon as I select an app for the embedded outpost it starts giving a 404 like @engAmirEng said.

I can try the debug stuff later.

[andres-asm]

This is my docker-compose.yml

services:
  postgresql:
    image: docker.io/library/postgres:12-alpine
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 5s
    volumes:
      - database:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: ${PG_PASS:?database password required}
      POSTGRES_USER: ${PG_USER:-authentik}
      POSTGRES_DB: ${PG_DB:-authentik}
    env_file:
      - .env
  redis:
    image: docker.io/library/redis:alpine
    command: --save 60 1 --loglevel warning
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    volumes:
      - redis:/data
  server:
    image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2024.2.4}
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
    volumes:
      - ./data/media:/media
      - ./data/templates:/templates
    env_file:
      - .env
    ports:
      - "${AUTHENTIK_PORT_HTTP:-9001}:9000"
      - "${AUTHENTIK_PORT_HTTPS:-9443}:9443"
    depends_on:
      - postgresql
      - redis
  worker:
    image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2024.2.4}
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
    # `user: root` and the docker socket volume are optional.
    # See more for the docker socket integration here:
    # https://goauthentik.io/docs/outposts/integrations/docker
    # Removing `user: root` also prevents the worker from fixing the permissions
    # on the mounted folders, so when removing this make sure the folders have the correct UID/GID
    # (1000:1000 by default)
    user: root
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./data/media:/media
      - ./data/certs:/certs
      - ./data/templates:/templates
    env_file:
      - .env
    depends_on:
      - postgresql
      - redis
  authentik_proxy:
    image: ghcr.io/goauthentik/proxy
    # Optionally specify which networks the container should be
    # might be needed to reach the core authentik server
    # networks:
    #   - foo
    ports:
        - 9009:9009
        - 9449:9443
    environment:
        AUTHENTIK_HOST: https://redacted
        AUTHENTIK_INSECURE: "false"
        AUTHENTIK_TOKEN: redacted
        # Starting with 2021.9, you can optionally set this too
        # when authentik_host for internal communication doesn't match the public URL
        # AUTHENTIK_HOST_BROWSER: https://external-domain.tld

volumes:
  database:
    driver: local
  redis:
    driver: local

[Schachte]

@andres-asm I'm using the embedded outpost just fine with the latest version. Please review my NGINX configuration and compare to yours. Note the headers. Maybe also post the UI config you have for the embedded outpost.

I got the manual outpost working as well, but the reverse proxy configuration between the two differs significantly (and it's not obvious). The docs are of course wrong.

[BeryJu]

To hopefully add some clarification to this:

This is not a bug, this is intended functionality. Previously, the /outpost.goauthentik.io paths would be available under all hostnames, not just the ones configured for the proxy providers. This meant that it was possible to go to the domain you access authentik under and to the path /outpost.goauthentik.io/start and start authentication and get an incorrect cookie. It would also happen quite frequently that with an incorrectly setup reverse proxy, that the reverse proxy would redirect to that invalid URL, which would cause redirect loops.

Hence since a couple releases, those URLs are only available when using the correct Host header to make sure this does not happen. Hence when accessing the URL directly in the browser on the authentik domain you see a 404 error, and when curling the /ping url you also get a 404 endpoint.

[Schachte]

@andres-asm @engAmirEng

I've updated my blog to walk through k8s setup. Working for me fine - https://ryan-schachte.com/notes/authentik_embedded_outpost_docker/

[ryanjaeb]

I've never used Authentik and was trying to configure it today. I think this is what's causing me confusion because I want to put Authenik on a dedicated (VM) host (auth.example.com) and have other hosts use it for auth. I'm not sure the docs take the behavior described here into account.

This is not a bug, this is intended functionality. Previously, the /outpost.goauthentik.io paths would be available under all hostnames, not just the ones configured for the proxy providers. This meant that it was possible to go to the domain you access authentik under and to the path /outpost.goauthentik.io/start and start authentication and get an incorrect cookie. It would also happen quite frequently that with an incorrectly setup reverse proxy, that the reverse proxy would redirect to that invalid URL, which would cause redirect loops.

Hence since a couple releases, those URLs are only available when using the correct Host header to make sure this does not happen. Hence when accessing the URL directly in the browser on the authentik domain you see a 404 error, and when curling the /ping url you also get a 404 endpoint.

Does that mean these docs are out of date?

If you're trying to proxy to an upstream over HTTPS, you need to set the Host header to the value they expect for it to work correctly.

reverse_proxy /outpost.goauthentik.io/* https://outpost.company {
    header_up Host {http.reverse_proxy.upstream.hostport}
}

I don't understand how that config would work anyway because it's saying to set the Host header to the upstream host for half of the proxied requests while leaving the Host as-is for the other half, isn't it? For now, assume that header_up Host would get set on both the reverse_proxy and forward_auth (docs), which is just an opinionated version of reverse_proxy. I think the config that page is describing would look like this:

app1.example.com {
        tls
        # Always forward outpost path to actual outpost
        reverse_proxy /outpost.goauthentik.io/* https://auth.example.com {
                # Host set to auth.example.com
                header_up Host {upstream_hostport}
        }

        # Forward authentication to outpost
        forward_auth https://auth.example.com {
                uri /outpost.goauthentik.io/auth/caddy
                header_up Host {upstream_hostport}
                copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
                trusted_proxies 192.0.2.1
        }

        reverse_proxy app1:80
}

If that's the case, I would only have to terminate TLS and proxy traffic for auth.example.com on my Authentik host:

Based on what @BeryJu said is the new behavior, you would not set the Host header because it would need to match the domain for the application in Authentik.

app1.example.com {
        tls
        # Always forward outpost path to actual outpost
        # Host defaults to app1.example.com
        reverse_proxy /outpost.goauthentik.io/* https://auth.example.com

        # Forward authentication to outpost
        forward_auth https://auth.example.com {
                uri /outpost.goauthentik.io/auth/caddy
                copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
                trusted_proxies 192.0.2.1
        }

        reverse_proxy app1:80
}

That would mean I need to do TLS termination for every domain on my Authentik host, right?

Is that correct?

[engAmirEng]

To hopefully add some clarification to this:

This is not a bug, this is intended functionality. Previously, the /outpost.goauthentik.io paths would be available under all hostnames, not just the ones configured for the proxy providers. This meant that it was possible to go to the domain you access authentik under and to the path /outpost.goauthentik.io/start and start authentication and get an incorrect cookie. It would also happen quite frequently that with an incorrectly setup reverse proxy, that the reverse proxy would redirect to that invalid URL, which would cause redirect loops.

Hence since a couple releases, those URLs are only available when using the correct Host header to make sure this does not happen. Hence when accessing the URL directly in the browser on the authentik domain you see a 404 error, and when curling the /ping url you also get a 404 endpoint.

what do you mean it's a feature????? then what is this? https://docs.goauthentik.io/docs/troubleshooting/forward_auth/general#ensure-outpostgoauthentikio-is-accessible

if they do not fix their doc there wont be a future for the project

[Schachte]

@BeryJu - I would agree the docs here are painfully convoluted and wasted many hours collectively between many folks in this thread, including myself. However, I at least debugged for a few days and shared a solution in multiple environments.

Would be great if someone could try it out for their use-case: https://ryan-schachte.com/notes/authentik_embedded_outpost_docker/

[Flying-Tom]

For me the following made it work again after almost eating my keyboard from frustration:

• go to Admin Area --> Applications --> Outposts --> authentik Embedded
• Actions -> Edit
• make sure your application is in the selected applications list
• 

after doing so it worked fine for me, hope this helps you.

It's been a while since I've last did stuff with the forward auth stuff but I don't think I remember that this was required earlier

It works for my caddy forward-auth! Thank you very much!

[ImDevinC]

I am getting this same problem on a brand new setup. Both the app I'm trying to login with and the /ping endpoint are returning a not found when using an embedded outpost. I've validated all the settings that everyone else has posted here, but still not working correctly. My setup in case it helps:

• Kubernetes
• nginx-ingress
• Embedded outpost

I've added the nginx annotations like so

    nginx.ingress.kubernetes.io/auth-response-headers: Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
    nginx.ingress.kubernetes.io/auth-signin: http://authentik.collins.home/outpost.goauthentik.io/start?rd=$escaped_request_uri
    nginx.ingress.kubernetes.io/auth-snippet: proxy_set_header X-Forwarded-Host $http_host;
    nginx.ingress.kubernetes.io/auth-url: http://ak-outpost-authentik-embedded-outpost.authentik.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx

Any insight is appreciated

[Schachte]

@ImDevinC - While I disagree with some of the documentation, did you try this? https://ryan-schachte.com/notes/authentik_embedded_outpost_docker/#kubernetes-setup

I got the latest version working no issues and tried to share resolution to a few folks

[ImDevinC]

@Schachte I have my configuration setup exactly as described in your document, which still looks to have issues. Also, unless I'm missing something, your documentation is the same as the Authentik Kubernetes documentation already, so no changes were needed

[Schachte]

@Schachte I have my configuration setup exactly as described in your document, which still looks to have issues. Also, unless I'm missing something, your documentation is the same as the Authentik Kubernetes documentation already, so no changes were needed

Can I assume the last line in your config is a typo on Github and not in your env? (Note the suffix)

 nginx.ingress.kubernetes.io/auth-url: http://authentik-server.authentik.svc.cluster.local/outpost.goauthentik.io/auth/ngi

Also, did you add the authentik_host in the Advanced settings panel and add the proper External host in the provider?

[ImDevinC]

Can I assume the last line in your config is a typo on Github and not in your env? (Note the suffix)

yeah, this was a bad copy/paste to Github but it is correct in my cluster

Also, did you add the authentik_host in the Advanced settings panel and add the proper External host in the provider?

Yes, the authentik_host was already configured when I went through the initial setup and I set the external host value to the proper URL for the service (in this case http://radarr.collins.home)

Update Got it working, here's what I had to change (I'll put together some documentation on this later)

1. In the Outpost section in Authentik, change the Integration to Local Kubernetes Cluster. I could've made this service/ingress manually, but if Authentik can manage it then why not. To validate this is working, you should see both an ingress and service named ak-outpost-authentik-embedded-outpost
2. In the Advanced Settings for this Outpost, make sure to set kubernetes_ingress_class_name: nginx
3. My annotations were in fact incorrect. I had the auth-url pointing to the main server, not the actual service. The annotations should read

   nginx.ingress.kubernetes.io/auth-response-headers: Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
   nginx.ingress.kubernetes.io/auth-signin: http://radarr.collins.home/outpost.goauthentik.io/start?rd=$escaped_request_uri
   nginx.ingress.kubernetes.io/auth-snippet: proxy_set_header X-Forwarded-Host $http_host;
   nginx.ingress.kubernetes.io/auth-url: http://ak-outpost-authentik-embedded-outpost.authentik.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx

Thanks for the help @Schachte

[Schachte]

Thanks for the help @Schachte

Awesome, glad to hear you got it working!