Kibana error 500 with outpost nginx reverse proxy

[KaoruKanon]

Describe your question With kibana protected by authk in auth revers proxy, I get a 500 error from kibana after a long load after authentication to sso

Relevant infos

For example, I have a jackett, homepage, radarr and sonarr protected by authentik that work as expected.

Authentik set up with a proxy Provider with a domain level Foward auth

The problem only concerns Kibana. Of course, Kibana worked without an issue before authentik was implemented. There must be something going on between authentik and kibana to explain this behavior.

Kibana is in minimum configuration. No SSL, no xpack, everything is default except for the Elasticsearch connection.

kibana nginx conf :

server {

    listen 443 ssl http2;
    listen       [::]:443 ssl http2;

    server_name kibana.domain.fr;

    index index.php index.html index.htm;

    access_log /var/log/nginx/access_https_kibana.domain.fr.log;
    error_log /var/log/nginx/error_https_kibana.domain.fr.log error;

    location / {
        proxy_pass http://tool.hosts.domain.fr:5601/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;

        ##############################
        # authentik-specific config
        ##############################
        proxy_buffers 8 16k;
        proxy_buffer_size 32k;
        auth_request     /outpost.goauthentik.io/auth/nginx;
        error_page       401 = @goauthentik_proxy_signin;
        auth_request_set $auth_cookie $upstream_http_set_cookie;
        add_header       Set-Cookie $auth_cookie;

        # translate headers from the outposts back to the actual upstream
        auth_request_set $authentik_username $upstream_http_x_authentik_username;
        auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
        auth_request_set $authentik_email $upstream_http_x_authentik_email;
        auth_request_set $authentik_name $upstream_http_x_authentik_name;
        auth_request_set $authentik_uid $upstream_http_x_authentik_uid;

        proxy_set_header X-authentik-username $authentik_username;
        proxy_set_header X-authentik-groups $authentik_groups;
        proxy_set_header X-authentik-email $authentik_email;
        proxy_set_header X-authentik-name $authentik_name;
        proxy_set_header X-authentik-uid $authentik_uid;

    }

    # all requests to /outpost.goauthentik.io must be accessible without authentication
    location /outpost.goauthentik.io {
        proxy_pass          https://sso.domain.fr/outpost.goauthentik.io;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Original-URL $scheme://$http_host$request_uri;
        add_header          Set-Cookie $auth_cookie;
        auth_request_set    $auth_cookie $upstream_http_set_cookie;
    }

    # Special location for when the /auth endpoint returns a 401,
    # redirect to the /start URL which initiates SSO
    location @goauthentik_proxy_signin {
        internal;
        add_header Set-Cookie $auth_cookie;
        return 302 /outpost.goauthentik.io/start?rd=$request_uri;
        # For domain level, use the below error_page to redirect to your authentik server with the full redirect path
        # return 302 https://authentik.company/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
    }

    ssl_certificate     /etc/letsencrypt/live/domain.fr/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/domain.fr/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;

}

Screenshots

Logs

nginx error :

2024/02/21 22:33:02 [error] 53723#53723: *43461 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.23.1, server: kibana.domain.fr, request: "POST /api/core/capabilities HTTP/2.0", subrequest: "/outpost.goauthentik.io/auth/nginx", upstream: "https://192.168.23.128:443/outpost.goauthentik.io/auth/nginx", host: "kibana.domain.fr", referrer: "https://kibana.domain.fr/app/home"
2024/02/21 22:33:02 [error] 53723#53723: *43461 auth request unexpected status: 504 while sending to client, client: 192.168.23.1, server: kibana.domain.fr, request: "POST /api/core/capabilities HTTP/2.0", host: "kibana.domain.fr", referrer: "https://kibana.domain.fr/app/home"

Version and Deployment (please complete the following information):

• authentik version: [e.g. 2023.10.7]
• Deployment: docker-compose (postgres dedicated database)

The problem is around /api/core/capabilities. According to the element inspector and the logs, I get a 500 I can't find any more information about this part on the Internet. I tried to set up a /api/core/capabilities location to bypass authentik. I get a slightly different error but I think it should work without any workarounds.

[KaoruKanon]

I can bypass the error this way. The kibana home page is there, but the site is only half-operational. I end up with a new KO URI

  location /api/core/capabilities {
          proxy_pass          http://tool.hosts.kaoru.fr:5601/api/core/capabilities;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_redirect off;
      }

[BeryJu]

The nginx logs point to a connection time out, could this be a network related issue?

Aside from that this could be caused by the fact that IIRC kibana is an SPA, meaning that all the actual functionality in the browser happens through Javascript, and apps like that can often cause issues with forward authentication