path /outpost.goauthentik.io not found

[mpeirone]

Describe the bug After updating to the version 2024.2.2 the path /outpost.goauthentik.io is "not found"(404)

Before opening this issue, I performed the following test: I did two fresh installations(even without the first login) on my k3s the first one with version 2023.10.7 the second one with version 2024.2.2. I verified that I did not have custom ingress/ingressRoute In version 2023 navigating to the url authentik.domain.com/outpost.goauthentik.io the page is correct. In version 2024 I get an error message "Not Found"

Authentik is installed via helm and the only change in the configuration file is as follows:

2023:

ingress:
    # Specify kubernetes ingress controller class name
    ingressClassName: traefik
    enabled: true
    tls:
      - secretName: authentik-tls
        hosts:
          - authentik.domain.com
    hosts:
        # Specify external host name
        - host: authentik.domain.com
          paths:
              - path: "/"
                pathType: Prefix`

2024:

server:
  ingress:
    # Specify kubernetes ingress controller class name
      ingressClassName: traefik
      enabled: true
      tls:
        - secretName: authentik-tls
          hosts:
            - authentik.domain.com
      hosts:
          - authentik.domain.com
      pathType: Prefix
      paths:
          - "/"

To Reproduce Perform a clean installation of the 2 versions

Expected behavior the path /outpost.goauthentik.io the path returns an error message: Not Found

Screenshots 2023: 2024:

Logs 2024.2.2

{"auth_via": "unauthenticated", "domain_url": "authentik.domain.com", "event": "/outpost.goauthentik.io", "host": "authentik.domain.com", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 43, "remote": "192.168.1.17", "request_id": "747e398cff3e4db19b6b9a1ef55e28ca", "runtime": 108, "schema_name": "public", "scheme": "http", "status": 404, "timestamp": "2024-03-18T21:59:50.605024", "user": "", "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"}

Version and Deployment (please complete the following information):

• authentik version: 2024.2.2
• Deployment: helm on K3s

Additional context Add any other context about the problem here.

[camrossi]

I had the same exact issue... turned out to be, for me 2 things:

1. All the guides are saying to create a new middleware however it seems one is already created by default for the embedded-outpost also the address needs to point to the local Kubernetes service i.e. http://ak-outpost-authentik-embedded-outpost.authentik:9000/outpost.goauthentik.io/auth/traefik this is already correct in the pre-configured one.

   kubectl -n authentik get middlewares.traefik.io
   NAME                                    AGE
   ak-outpost-authentik-embedded-outpost   26h

2. Since at the beginning I made a mistake and put my external URL in a Middleware I created the browser had something cached and was stuck on going to the wrong URL...even after changing to the K8s local URL. I tested with a different browser and it worked fine, i eventually cleared all the cookies and application data from Chrome and also Chrome started to work.

   I confirm I am on 2024.2.2

[mpeirone]

Unfortunately, I have no embedded-outpost. Did you install it using helm? It would be possible to see your helm values? What answer do you get if you query the page yourpublic.dns/outpost.goauthentik.io ? Thanks

[mpeirone]

I don't think it's a middleware-related problem, both forward auth and mypublic.dns/outpost.goauthentik.io/ page respond 404. The problem is very similar to #8932

[camrossi]

I had exactly that same page as well, that was caused by some missconfig I had, when I was looking at the logs in traefik I was getting error saying my certificates where missing and indeed they where. I would suggest checking if traefik can resolve all the services and has no errors in its logs.

That is how I eventually sorted it out. At least in my case.

As for my configs are all on github just check my home-cluster repo :)

[AryanOvalekar]

Have the same issue on the docker version, did 2 fresh install tests, one with 2024.2.2, another with 2023.10.7, I was able to access /outpost.goauthentik.io on the 10.7 version, but on the 2.2 I received the same not found page. This was on a fresh docker install with no other containers, just the docker-compose stack from the installation page. Edit: Forgot to mention that this is entirely using the embedded outpost, and with no applications or providers, although adding an application and provider did not change the results.

[anleg]

Hey 👋 I have the exact same issue after moving from 2023.10 to 2014.2. After checking the changelog, I'm wandering if this PR (https://github.com/goauthentik/authentik/pull/7539/files) could have caused the issue.

[fREAST]

Also had this issue after upgrading, and managed to resolve it by changing the redirect location on the provider.

The provider was always configured with "Forward auth (single application)" mode. The old configuration was working fine until I upgraded to 2024.2.2.

This was the old configuration in nginx for the app.

location @goauthentik_proxy_signin {
  internal;
  add_header Set-Cookie $auth_cookie;
  #return 302 /outpost.goauthentik.io/start?rd=$request_uri;
  return 302 https://authentik.example.com/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
}

And when I browsed https://myapp.example.com/, I was obviously redirected to https://authentik.example.com/outpost.goauthentik.io/start?rd=https://myapp.example.com/ and now it was only showing the "Not Found" page.

The fix was as simple as changing the redirect to return 302 /outpost.goauthentik.io/start?rd=$request_uri; instead (its commented above), and now it properly redirects to the authentik login page.

curl -vL https://myapp.example.com

location: https://myapp.example.com/outpost.goauthentik.io/start?rd=/

location: https://authentik.example.com/application/o/authorize/?client_id=

I know it says in the example configuration, this should be used for domain level providers, and not single application.

# For domain level, use the below error_page to redirect to your authentik server with the full redirect path
# return 302 https://authentik.company/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;

Not sure if that was changed recently, but I have always had the redirect configued as above, and it was working fine prior to upgrading, maybe as @anleg suggested, related to the PR #7539.

[GGG-KILLER]

Have the same issue, just downloaded the docker compose example, ran it, tried to do curl -vso /dev/null http://localhost:9000/outpost.goauthentik.io/ping and get a 404.

I also set up https://example.com as an application with Forward Mode proxying and then attempted to do curl -vso /dev/null 'http://localhost:9000/outpost.goauthentik.io/start?rd=https%3A%2F%2Fexample.com', which also returned 404.

This makes the builtin proxy totally broken.

[GGG-KILLER]

Update: my issue was the same as @fREAST, just by changing my command a bit (note the part setting the Host: header, it's important) I started getting the 302 response:

$ curl -kvso /dev/null -H 'Host: example.com' -H 'X-Original-URL: https://example.com/' 'https://sso.shiro.lan/outpost.goauthentik.io/start?rd=https%3A%2F%2Fexample.com'
* Host sso.shiro.lan:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.2.247
*   Trying 192.168.2.247:443...
* Connected to sso.shiro.lan (192.168.2.247) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1016 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=sso.shiro.lan
*  start date: Apr  3 19:09:44 2024 GMT
*  expire date: Apr  4 19:10:44 2024 GMT
*  issuer: O=StepCA.Lan; CN=StepCA.Lan Intermediate CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://sso.shiro.lan/outpost.goauthentik.io/start?rd=https%3A%2F%2Fexample.com
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: example.com]
* [HTTP/2] [1] [:path: /outpost.goauthentik.io/start?rd=https%3A%2F%2Fexample.com]
* [HTTP/2] [1] [user-agent: curl/8.6.0]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [x-original-url: https://example.com/]
} [5 bytes data]
> GET /outpost.goauthentik.io/start?rd=https%3A%2F%2Fexample.com HTTP/2
> Host: example.com
> User-Agent: curl/8.6.0
> Accept: */*
> X-Original-URL: https://example.com/
> 
{ [5 bytes data]
< HTTP/2 302 
< server: nginx
< date: Wed, 03 Apr 2024 21:59:03 GMT
< content-type: text/html; charset=utf-8
< content-length: 350
< location: https://sso.shiro.lan/application/o/authorize/?client_id=...&redirect_uri=...&response_type=code&scope=email+ak_proxy+profile+openid&state=...
< set-cookie: authentik_proxy_6MQSP9KS=...; Path=/; Expires=Thu, 04 Apr 2024 21:59:04 GMT; Max-Age=86401; HttpOnly; Secure; SameSite=Lax
< vary: Accept-Encoding
< 
{ [350 bytes data]
* Connection #0 to host sso.shiro.lan left intact

[MattZera]

I have the same issue. I'm running k8s with an nginx ingress.

The documentation needs to be updated to support the host header requirement if its possible.

After about an hour of googling and using chatgpt, I'm unable to get this working with an nginx ingress even though I know that setting the host header should resolve it.

this is especially frustrating because the nginx ingress documentation specially mentions using the internal cluster dns because nginx might rewrite headers. I'm not an expert, but I think this is incompatible with needing a host header to be set because the host will never match "pod-name.namespace.svc.cluster.local:9000"

[P4R4DiSi4C]

Same issue here if I try to ping on version 2024.2.3 I do get a 302 and authentication it still does work with an app that I've setup behind NGINX Proxy Manager but this doesn't work: https://docs.goauthentik.io/docs/troubleshooting/forward_auth/general#ensure-outpostgoauthentikio-is-accessible

curl -v https://app.company/outpost.goauthentik.io/ping

[GGG-KILLER]

Same issue here if I try to ping on version 2024.2.3 I do get a 302 and authentication it still does work with an app that I've setup behind NGINX Proxy Manager but this doesn't work: https://docs.goauthentik.io/docs/troubleshooting/forward_auth/general#ensure-outpostgoauthentikio-is-accessible

curl -v https://app.company/outpost.goauthentik.io/ping

@P4R4DiSi4C the /ping endpoint doesn't exist (even though the docs talk about it), what you need to do is this to test if it works:

$ curl -vso /dev/null 'https://app.company/outpost.goauthentik.io/start?rd=/'

[BeryJu]

This very much looks like the embedded outpost isn't start for some reason, please post the server container's logs

@GGG-KILLER the /ping endpoint does exist, however if the embedded outpost is not running it won't return anything

[GGG-KILLER]

@GGG-KILLER the /ping endpoint does exist, however if the embedded outpost is not running it won't return anything

@BeryJu If it does exist, it doesn't seem to be working on my instance at all. The following return 404:

$ curl -vso /dev/null -H 'Host: mega.shiro.lan' 'https://sso.shiro.lan/outpost.goauthentik.io/ping'
< HTTP/2 404 
< server: nginx
< date: Tue, 23 Apr 2024 16:44:10 GMT
< content-type: text/plain; charset=utf-8
< content-length: 19
< vary: Accept-Encoding
< x-content-type-options: nosniff
< 
{ [19 bytes data]
$ curl -vso /dev/null 'https://sso.shiro.lan/outpost.goauthentik.io/ping'
< HTTP/2 404 
< server: nginx
< date: Tue, 23 Apr 2024 16:45:10 GMT
< content-type: text/html; charset=utf-8
< content-length: 3568
< vary: Accept-Encoding
< referrer-policy: same-origin
< vary: Accept-Encoding
< vary: Cookie
< x-authentik-id: 05ebd09f661a426b9aa9dcb187918109
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-powered-by: authentik
< 

However the /start endpoint works perfectly:

$ curl -vso /dev/null -H 'Host: mega.shiro.lan' 'https://sso.shiro.lan/outpost.goauthentik.io/start?rd=/'
< HTTP/2 302 
< server: nginx
< date: Tue, 23 Apr 2024 16:45:44 GMT
< content-type: text/html; charset=utf-8
< content-length: 354
< location: https://sso.shiro.lan/application/o/authorize/?client_id=...&redirect_uri=...&response_type=code&scope=openid+profile+ak_proxy+email&state=...
< set-cookie: authentik_proxy_4i6JRZsF=...; Path=/; Expires=Wed, 24 Apr 2024 16:45:45 GMT; Max-Age=86401; HttpOnly; Secure; SameSite=Lax
< vary: Accept-Encoding
< 

[BeryJu]

The /ping endpoint is actually on a different endpoint, it's hosted on the :9300 endpoint which has prometheus metrics too, so the docs will need to be updated for that

[ilijamt]

I am also experiencing the same issue. This worked on 2023 on the same k3s cluster, and after upgrade to 2024 it stopped working. Installation was done using helm. Kubernetes version: v1.29.4

[ekrekeler]

Have the same issue on the docker version, did 2 fresh install tests, one with 2024.2.2, another with 2023.10.7, I was able to access /outpost.goauthentik.io on the 10.7 version, but on the 2.2 I received the same not found page. This was on a fresh docker install with no other containers, just the docker-compose stack from the installation page. Edit: Forgot to mention that this is entirely using the embedded outpost, and with no applications or providers, although adding an application and provider did not change the results.

I am also having the same problem with my existing docker compose installation. On 2023.10.7, the embedded outpost will respond correctly to the healthchecks. Here is curl run from inside the authentik-server container:

root@e1df5ee23dc8:/# curl -vs -o /dev/null http://localhost:9000/outpost.goauthentik.io/ping
*   Trying 127.0.0.1:9000...
* Connected to localhost (127.0.0.1) port 9000 (#0)
> GET /outpost.goauthentik.io/ping HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 204 No Content
< Vary: Accept-Encoding
< Date: Sun, 05 May 2024 19:48:08 GMT
<
* Connection #0 to host localhost left intact

And same test after upgrading to 2024.4.1:

root@215946beeb44:/# curl -vs -o /dev/null http://localhost:9000/outpost.goauthentik.io/ping
*   Trying 127.0.0.1:9000...
* Connected to localhost (127.0.0.1) port 9000 (#0)
> GET /outpost.goauthentik.io/ping HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Length: 3568
< Content-Type: text/html; charset=utf-8
< Date: Sun, 05 May 2024 20:00:42 GMT
< Referrer-Policy: same-origin
< Vary: Accept-Encoding
< Vary: Cookie
< X-Authentik-Id: d4a245f92c4e4486b40413d4a82ed5d9
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Powered-By: authentik
<
{ [3568 bytes data]
* Connection #0 to host localhost left intact

Everything else works normally. The embedded outpost is up and serving authentication requests.

I also tested with curl to http://localhost:9300/outpost.goauthentik.io/ping but this returned HTTP 404 on both versions.

[xMinhx]

Have the same issue. If I try to access /outpost.goauthentik.io, I also get the error page not found 404. Embedded outpost is online and seems to be working properly. Same if I try to access it through port 9300.

Authentik version 2024.4.2

Edit: Tested with version: 2023.10.4: Working 2023.10.7: Working 2024.2.0-rc1: Not working 2024.2.2: Not working

Changes somewhere between 2024.2.0-rc1 and 2023.10.7 seems to be the issue

[evilprophet]

Today I spotted that I have the same issue.

Version: 2024.4.2, run inside docker.

[Clear2437]

Same issue here, clean fresh docker compose install with no settings changed. The embedded outpost is not working on 2024.4.2 and also not working on the newest version 2024.6.0-rc2. Not found page.

In my server logs im getting these errors:

warning | error=authentik starting event=failed to proxy to backend logger=authentik.router timestamp=2024-06-22T11:26:49Z

INF | event=Starting gunicorn 22.0.0 logger=gunicorn.error timestamp=1719055628.460975

INF | event=Listening at: unix:/dev/shm/authentik-core.sock (17) logger=gunicorn.error timestamp=1719055628.461775

INF | event=Using worker: lifecycle.worker.DjangoUvicornWorker logger=gunicorn.error timestamp=1719055628.4618392

INF | event=Booting worker with pid: 54 logger=gunicorn.error timestamp=1719055628.4681582

INF | event=Booting worker with pid: 55 logger=gunicorn.error timestamp=1719055628.5095584

ERR | error=403 Forbidden event=Failed to fetch outpost configuration, retrying in 3 seconds logger=authentik.outpost.ak-api-controller timestamp=2024-06-22T11:27:08Z

[seraphblade2010]

Is there any movement on this? Or some workaround to get this working? Currently trying to have longhorn behind authentik auth but this bug prevents it :( Suffering from the same problem, k3s deployment with traefik and authentik helm charts with the same values as the author of this issue.

[plsnotracking]

Helm deployment: 2024.6.1

The one odd thing I saw on my Authentik dashboard was under System > Outpost Integrations > Local Kubernetes Cluster was Unhealthy but on editing and saving it'll temporary be happy, and fail again once page is refreshed. I have no idea exactly what is even unhealthy, since other Auth works fine except for Proxy.

Screenshot

Logs from server service:

{"event":"/outpost.goauthentik.io/auth/nginx","host":"app.enterprise.com","level":"info","logger":"authentik.outpost.proxyv2.application","method":"GET","name":"app","remote":"10.42.0.19","runtime":"4.71
6","scheme":"http","size":21,"status":401,"timestamp":"2024-07-18T07:25:47Z","user_agent":"Mozilla/5.0 (Windows NT 10.0; rv:128.0) Gecko/20100101 Firefox/128.0"}
{"auth_via": "unauthenticated", "domain_url": "auth.enterprise.com", "event": "/outpost.goauthentik.io/start?rd=https://app.enterprise.com/", "host": "auth.enterprise.com", "level": "info", "logger": "
authentik.asgi", "method": "GET", "pid": 46, "remote": "10.42.2.28", "request_id": "d1d62aa5e6e248b9a3fe6b723bd25285", "runtime": 4, "schema_name": "public", "scheme": "https", "status": 404, "timestamp": "2024-07-
18T07:25:47.510619", "user": "", "user_agent": "Mozilla/5.0 (Windows NT 10.0; rv:128.0) Gecko/20100101 Firefox/128.0"}

Logs from the outpost service:

{"event":"/outpost.goauthentik.io/auth/nginx","host":"app.enterprise.com","level":"info","logger":"authentik.outpost.proxyv2.application","method":"GET","name":"app","remote":"10.42.0.19","runtime":"4.71
6","scheme":"http","size":21,"status":401,"timestamp":"2024-07-18T07:25:47Z","user_agent":"Mozilla/5.0 (Windows NT 10.0; rv:128.0) Gecko/20100101 Firefox/128.0"}
{"auth_via": "unauthenticated", "domain_url": "app.enterprise.com", "event": "/outpost.goauthentik.io/start?rd=https://app.enterprise.com/", "host": "app.enterprise.com", "level": "info", "logger": "
authentik.asgi", "method": "GET", "pid": 46, "remote": "10.42.2.28", "request_id": "d1d62aa5e6e248b9a3fe6b723bd25285", "runtime": 4, "schema_name": "public", "scheme": "https", "status": 404, "timestamp": "2024-07-
18T07:25:47.510619", "user": "", "user_agent": "Mozilla/5.0 (Windows NT 10.0; rv:128.0) Gecko/20100101 Firefox/128.0"}

Here's what my ingress looks like

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  namespace: database
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-url: http://ak-outpost-authentik-embedded-outpost.authentik.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx
    nginx.ingress.kubernetes.io/auth-signin: https://auth.enterprise.com/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri
    nginx.ingress.kubernetes.io/auth-response-headers: Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
    nginx.ingress.kubernetes.io/auth-snippet: |
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;  # Forward real client IP
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  # Forward client IPs
      proxy_set_header X-Forwarded-Proto $scheme;  # Forward protocol (HTTP/HTTPS)
spec:
  rules:
  - host: app.enterprise.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: app
              port:
                number: 80

Results in

Please let me know if I can provide anymore data at all. Thanks.

[ilijamt]

Same for me after upgrade to 2024.6.1

[unbelauscht]

Please react with emojis or subscribe to the issue.

Bumping the thread just spams subscribed people.

[rlenferink]

For me (with Authentik 2024.6.3) this is still a problem. Reading through this thread the only workaround seems to be to downgrade to 2023.10.7, but I am afraid that there is no upgrade path after that.

Does anyone know if work is being done to resolve this issue?

[camrossi]

For me it works, check my comment from March. I had the same issue and what I wrote back then fixed it for me and have been working flawlessly ever since.

[rlenferink]

@camrossi thanks for you response! Unfortunately I don't have a Kubernetes environment running so I cannot test with the Kubernetes deployment steps.

I am using:

• Fedora 40
• docker-compose v2.29.1
• podman 5.1.1 (rootless)

For both versions I've tried the same URL: http://<authentik host>/outpost.goauthentik.io/start?rd=http://demo-app.loc:9999

When starting a clean environment with 2023.10.7 I can confirm the forward proxy works and the /output.goauthentik.io endpoint resolves correctly and redirects to the default-authentication-flow.

When starting a clean environment with 2024.6.3 and visiting the above URL, no redirect is executed and a 404 Not Found is returned instead.

[plsnotracking]

@camrossi would it be possible to link the comment/issue? Cursory search with your username did not return a good deal about this. I'm experiencing very similar issues to @rlenferink, old envs are fine, new envs run into 404. Thanks.

[camrossi]

@plsnotracking sure here: https://github.com/goauthentik/authentik/issues/8956#issuecomment-2007162929

Also @rlenferink I run this on my "home" K8s clusters and have never tested on anything else than this one environment but I recall the struggle was quite long with this one :(

[davispuh]

This issue is still present with latest 2024.6.3 Docker image.

I'm using Podman Quadlets with rootless containers

$ cat ~/.config/containers/systemd/Authentik-Server.container

[Unit]
Description=Authentik Server container
After=local-fs.target

[Container]
Image=ghcr.io/goauthentik/server:latest
Exec=server
EnvironmentFile=/var/lib/authentik/.config/containers/systemd/Authentik.env
Network=slirp4netns:allow_host_loopback=true
PublishPort=127.0.0.1:19000:9000
PublishPort=127.0.0.1:19300:9300
UserNS=keep-id:uid=1000,gid=1000
Volume=/var/lib/authentik/media:/media
Volume=/var/lib/authentik/templates:/templates
AutoUpdate=registry

[Install]
WantedBy=multi-user.target default.target

$ cat ~/.config/containers/systemd/Authentik-Worker.container

[Unit]
Description=Authentik Worker container
After=local-fs.target

[Container]
Image=ghcr.io/goauthentik/server:latest
Exec=worker
EnvironmentFile=/var/lib/authentik/.config/containers/systemd/Authentik.env
Network=slirp4netns:allow_host_loopback=true
UserNS=keep-id:uid=1000,gid=1000
Volume=/var/lib/authentik/media:/media
Volume=/var/lib/authentik/templates:/templates
Volume=/var/lib/authentik/certs:/certs
AutoUpdate=registry

[Install]
WantedBy=multi-user.target default.target

# systemctl --user --machine=authentik@ daemon-reload
# systemctl --user --machine=authentik@ start Authentik-Server
# systemctl --user --machine=authentik@ start Authentik-Worker

Not using Docker integration so there isn't any.

Outpost shows as working

And documentation at https://version-2024-6.goauthentik.io/docs/troubleshooting/forward_auth/general#ensure-outpostgoauthentikio-is-accessible is correct

$ curl -v https://auth.example.org/outpost.goauthentik.io/ping

Should return HTTP 204 but instead we get 404.

Workaround is deploying separate Proxy Outpost instance.

https://version-2024-6.goauthentik.io/docs/outposts/manual-deploy-docker-compose

[engAmirEng]

any update on this? issue still persists in 2024.6.3

[sergeykorablin]

It seems like the developers don't care about this problem

[ilijamt]

So looking at the configuration that Authentik Generates for me:

# Upgrade WebSocket if requested, otherwise use keepalive
map $http_upgrade $connection_upgrade_keepalive {
    default upgrade;
    ''      '';
}

server {
    # SSL and VHost configuration
    listen                  443 ssl http2;
    server_name             _;

    ssl_certificate         /etc/ssl/certs/ssl-cert-snakeoil.pem;
    ssl_certificate_key     /etc/ssl/private/ssl-cert-snakeoil.key;

    # Increase buffer size for large headers
    # This is needed only if you get 'upstream sent too big header while reading response
    # header from upstream' error when trying to access an application protected by goauthentik
    proxy_buffers 8 16k;
    proxy_buffer_size 32k;

    location / {
        # Put your proxy_pass to your application here, and all the other statements you'll need
        # proxy_pass http://localhost:5000;
        # proxy_set_header Host $host;
        # proxy_set_header ...
        # Support for websocket
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade_keepalive;

        ##############################
        # authentik-specific config
        ##############################
        auth_request     /outpost.goauthentik.io/auth/nginx;
        error_page       401 = @goauthentik_proxy_signin;
        auth_request_set $auth_cookie $upstream_http_set_cookie;
        add_header       Set-Cookie $auth_cookie;

        # translate headers from the outposts back to the actual upstream
        auth_request_set $authentik_username $upstream_http_x_authentik_username;
        auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
        auth_request_set $authentik_email $upstream_http_x_authentik_email;
        auth_request_set $authentik_name $upstream_http_x_authentik_name;
        auth_request_set $authentik_uid $upstream_http_x_authentik_uid;

        proxy_set_header X-authentik-username $authentik_username;
        proxy_set_header X-authentik-groups $authentik_groups;
        proxy_set_header X-authentik-email $authentik_email;
        proxy_set_header X-authentik-name $authentik_name;
        proxy_set_header X-authentik-uid $authentik_uid;
    }

    # all requests to /outpost.goauthentik.io must be accessible without authentication
    location /outpost.goauthentik.io {
        proxy_pass              http://outpost.company:9000/outpost.goauthentik.io;
        # ensure the host of this vserver matches your external URL you've configured
        # in authentik
        proxy_set_header        Host $host;
        proxy_set_header        X-Original-URL $scheme://$http_host$request_uri;
        add_header              Set-Cookie $auth_cookie;
        auth_request_set        $auth_cookie $upstream_http_set_cookie;
        proxy_pass_request_body off;
        proxy_set_header        Content-Length "";
    }

    # Special location for when the /auth endpoint returns a 401,
    # redirect to the /start URL which initiates SSO
    location @goauthentik_proxy_signin {
        internal;
        add_header Set-Cookie $auth_cookie;
        return 302 /outpost.goauthentik.io/start?rd=$request_uri;
        # For domain level, use the below error_page to redirect to your authentik server with the full redirect path
        # return 302 https://authentik.company/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
    }
}

So looking at some of the urls.

• http://outpost.company:9000
• https://authentik.company

❯ k get ingress
NAME                                    CLASS    HOSTS                  ADDRESS                         PORTS     AGE
authentik-server                        nginx    authentik.example.com   10.0.50.1,10.0.60.1,10.0.60.2   80, 443   106d
ak-outpost-authentik-embedded-outpost   <none>   uptime.example.com                                      80, 443   106d

This is what I have for ingress.

And running

❯ curl https://authentik.example.com/outpost.goauthentik.io -I
HTTP/2 404
date: Sun, 18 Aug 2024 21:03:34 GMT
content-type: text/html; charset=utf-8
content-length: 3501
referrer-policy: same-origin
vary: Accept-Encoding
vary: Cookie
x-authentik-id: 43ec28ade2ab4b52acac9915a8019f47
x-content-type-options: nosniff
x-frame-options: DENY
x-powered-by: authentik
strict-transport-security: max-age=31536000; includeSubDomains

This is what I see in the app, that authentik responds with a 404 and nginx gives me an error 500.

❯ k get all
NAME                                    READY   STATUS    RESTARTS   AGE
pod/authentik-redis-master-0            1/1     Running   0          58d
pod/authentik-worker-549968b974-f2p9d   1/1     Running   0          15d
pod/authentik-server-65f5958778-8sdmm   1/1     Running   0          15d
pod/authentik-server-65f5958778-rj86z   1/1     Running   0          15d
pod/authentik-worker-549968b974-kx2kq   1/1     Running   0          15d

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/authentik-redis-headless                ClusterIP   None            <none>        6379/TCP                     106d
service/authentik-redis-master                  ClusterIP   10.43.196.134   <none>        6379/TCP                     106d
service/authentik-server                        ClusterIP   10.43.164.185   <none>        80/TCP,443/TCP               106d
service/ak-outpost-authentik-embedded-outpost   ClusterIP   10.43.165.216   <none>        9000/TCP,9300/TCP,9443/TCP   106d

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/authentik-server   2/2     2            2           106d
deployment.apps/authentik-worker   2/2     2            2           106d

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/authentik-server-65f5958778   2         2         2       15d
replicaset.apps/authentik-worker-549968b974   2         2         2       15d

NAME                                      READY   AGE
statefulset.apps/authentik-redis-master   1/1     100d

And the service for the embedded output is

❯ k describe service/ak-outpost-authentik-embedded-outpost
Name:              ak-outpost-authentik-embedded-outpost
Namespace:         authentik
Labels:            app.kubernetes.io/instance=authentik-embedded-outpost
                   app.kubernetes.io/managed-by=goauthentik.io
                   app.kubernetes.io/name=authentik-proxy
                   app.kubernetes.io/version=2024.6.2
                   goauthentik.io/outpost-name=authentik-embedded-outpost
                   goauthentik.io/outpost-type=proxy
                   goauthentik.io/outpost-uuid=852db20c37d0457e8a30f7a3b5201d92
Annotations:       <none>
Selector:          app.kubernetes.io/component=server,app.kubernetes.io/name=authentik
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.165.216
IPs:               10.43.165.216
Port:              http  9000/TCP
TargetPort:        9000/TCP
Endpoints:         10.42.1.17:9000,10.42.2.116:9000
Port:              http-metrics  9300/TCP
TargetPort:        9300/TCP
Endpoints:         10.42.1.17:9300,10.42.2.116:9300
Port:              https  9443/TCP
TargetPort:        9443/TCP
Endpoints:         10.42.1.17:9443,10.42.2.116:9443
Session Affinity:  None
Events:            <none>

And the outpost ingress

❯ k describe ingress ak-outpost-authentik-embedded-outpost
Name:             ak-outpost-authentik-embedded-outpost
Labels:           app.kubernetes.io/instance=authentik-embedded-outpost
                  app.kubernetes.io/managed-by=goauthentik.io
                  app.kubernetes.io/name=authentik-proxy
                  app.kubernetes.io/version=2024.6.2
                  goauthentik.io/outpost-name=authentik-embedded-outpost
                  goauthentik.io/outpost-type=proxy
                  goauthentik.io/outpost-uuid=852db20c37d0457e8a30f7a3b5201d92
Namespace:        authentik
Address:
Ingress Class:    <none>
Default backend:  <default>
TLS:
  authentik-outpost-tls terminates uptime.example.com
Rules:
  Host               Path  Backends
  ----               ----  --------
  uptime.example.com
                     /outpost.goauthentik.io   ak-outpost-authentik-embedded-outpost:http (10.42.1.17:9000,10.42.2.116:9000)
Annotations:         nginx.ingress.kubernetes.io/affinity: cookie
                     nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
                     nginx.ingress.kubernetes.io/proxy-buffers-number: 4
                     traefik.ingress.kubernetes.io/affinity: true
                     traefik.ingress.kubernetes.io/router.tls: true
Events:              <none>

Looking at the ingress for authentik

❯ k describe ingress authentik-server
Name:             authentik-server
Labels:           app.kubernetes.io/component=server
                  app.kubernetes.io/instance=authentik
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=authentik
                  app.kubernetes.io/part-of=authentik
                  app.kubernetes.io/version=2024.6.2
                  helm.sh/chart=authentik-2024.6.2
                  helm.toolkit.fluxcd.io/name=authentik
                  helm.toolkit.fluxcd.io/namespace=authentik
Namespace:        authentik
Address:          10.0.50.1,10.0.60.1,10.0.60.2
Ingress Class:    nginx
Default backend:  <default>
TLS:
  authentik.example.com-tls terminates authentik.example.com
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  authentik.example.com
                        /   authentik-server:80 (10.42.1.17:9000,10.42.2.116:9000)
Annotations:            cert-manager.io/cluster-issuer: letsencrypt-prod
                        gethomepage.dev/description: The authentication glue you need.
                        gethomepage.dev/enabled: true
                        gethomepage.dev/group: Infrastructure
                        gethomepage.dev/icon:
                          https://camo.githubusercontent.com/f068956b02827bfce3b652ac3cbbe563ca05294aa149f73d1049059624445b37/68747470733a2f2f676f61757468656e74696b...
                        gethomepage.dev/name: Authentik
                        kubernetes.io/tls-acme: true
                        meta.helm.sh/release-name: authentik
                        meta.helm.sh/release-namespace: authentik
Events:                 <none>

And looking at the ingress-nginx logs it seems that it's going to authentik-server pods. In the nginx logs I can only see.

x.x.x.x - - [18/Aug/2024:21:09:26 +0000] "GET /outpost.goauthentik.io HTTP/2.0" 404 3501 "-" "curl/8.6.0" 52 0.056 [authentik-authentik-server-80] [] 10.42.2.116:9000 3501 0.056 404 f6e177a2e7f28cce509de69f5beb0e35

@BeryJu where does this need to route? To the server pods? or to the ak-outpost-authentik-embedded-outpost ingress and if so how does it need to be routed there? It seems as I'm missing some glue here to connect it to the correct location.

What should http://outpost.company:9000 point to?

[Schachte]

What should http://outpost.company:9000 point to?

@ilijamt This needs to point to your authentik server. If you're running in Docker-Compose for example, this could be the service name of the server like authentik or server. K8s would be similar.

If you're using the embedded outpost not the manual outpost, then:

        location /outpost.goauthentik.io {
            proxy_pass              http://server:9000/outpost.goauthentik.io;

[ilijamt]

That’s what I have it point to. But it does not work. The service I’m using is uptime Kuma is external and not inside the k8s cluster.

So before the upgrade to 2024 this worked without an issue. The proxy pass pointed to https://authentik.example.com. But since 2024 version it doesn’t work. I use k8s with nginx and not traefik. So something has changed between the last version of 2023 and 2024.

Is there a way to troubleshoot the flow to find outs why the embedded outpost does not work? As you can see in the text it goes to the server deployment but it returns a 404.

On Mon, 19 Aug 2024 at 00:47, Ryan Schachte @.***> wrote:

What should http://outpost.company:9000 point to?

@ilijamt https://github.com/ilijamt This needs to point to your authentik server. If you're running in Docker-Compose for example, this could be the service name of the server like authentik or server. K8s would be similar.

If you're using the embedded outpost not the manual outpost, then:

    location /outpost.goauthentik.io {
        proxy_pass              http://server:9000/outpost.goauthentik.io;

— Reply to this email directly, view it on GitHub https://github.com/goauthentik/authentik/issues/8956#issuecomment-2295422283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIWFY5Y3CAGPL4L5WQEBJDZSEQBVAVCNFSM6AAAAABE4M72OKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVGQZDEMRYGM . You are receiving this because you were mentioned.Message ID: @.***>

[Schachte]

I'm not using Kubernetes, so it's hard for me to suggest specifically, but a good start would be the logs for Authentik.

I would also add more granular debug logs on NGINX side to see what request headers are flowing through the system.

For example, ❯ curl https://authentik.example.com/outpost.goauthentik.io -I

Why would you curl this mate? This is not a real URL. authentik.example.com won't resolve to anything from a DNS perspective since this is just an example domain.

[Schachte]

Added notes for getting this working with docker-compose and NGINX. https://ryan-schachte.com/notes/authentik_embedded_outpost_docker/

[ilijamt]

If you look at the comment I made you can see the output of the curl command as well as the corresponding nginx log.

On Mon, 19 Aug 2024 at 01:01, Ryan Schachte @.***> wrote:

I'm not using Kubernetes, so it's hard for me to suggest specifically, but a good start would be the logs for Authentik.

I would also add more granular debug logs on NGINX side to see what request headers are flowing through the system.

For example, ❯ curl https://authentik.example.com/outpost.goauthentik.io -I

Why would you curl this mate? This is not a real URL. authentik.example.com won't resolve to anything from a DNS perspective since this is just an example domain.

— Reply to this email directly, view it on GitHub https://github.com/goauthentik/authentik/issues/8956#issuecomment-2295425571, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIWFYZM3QMHTPLP5C5NDF3ZSERTTAVCNFSM6AAAAABE4M72OKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVGQZDKNJXGE . You are receiving this because you were mentioned.Message ID: @.***>

[Schachte]

❯ curl https://authentik.example.com/outpost.goauthentik.io -I

Ah I see, you configured this dummy host. Are you trying to use embedded outpost or no? Your NGINX conf shows external outpost when it should point to authentik core itself. I would think you would want authentik.example.com, but check my blog post above and lmk.

        proxy_pass              http://outpost.company:9000/outpost.goauthentik.io;

[jlssmt]

same problem here

for me even setup route throws 404 when accessing after clean install via reverse-proxy and tld: https://auth.example.com/if/flow/initial-setup/

route/setup is working when accessing container directly: http://192.168.0.101:9000/if/flow/initial-setup/

so this has to be a misconfigured nginx i'm using swag (lsio) and their defaults.