NGINX auth_request 500 internal server error

[JantsoP]

Describe your question/ I will reopen issue #2594 due it has been closed for months now and there is no proper answer on how to fix it. I have ran to this exact same problem. Issue is that when running config provided by authentik + added my HTTP->HTTPS redirect and actual internal app which it needs to call after auth, I do get error 500 on browser, 502 in logs.

Relevant infos I am running NGINX as standalone and I have configured reverse proxy per Authentik documentation. Latest Authentik in use which Docker can provide on Hetzner VM.

domain1.tld - Authentik server running on different domain, domain2.tld - Domain which has the actual service running.

Authentik NGINX config:

# Upstream where your authentik server is hosted.
upstream authentik {
    server localhost:9443;
    # Improve performance by keeping some connections alive.
    keepalive 10;
}

# Upgrade WebSocket if requested, otherwise use keepalive
map $http_upgrade $connection_upgrade_keepalive {
    default upgrade;
    ''      '';
}

server {
    # HTTP server config
    listen 80;
    server_name auth.domain1.tld;

    # 301 redirect to HTTPS
    location / {
            return 301 https://$host$request_uri;
    }
}
server {
    # HTTPS server config
    listen 443 ssl http2;
    server_name auth.domain1.tld;

    # TLS certificates
    ssl_certificate /var/ssl/domain1/fullchain.pem;
    ssl_certificate_key /var/ssl/domain1/privkey.pem;

    # Proxy site
    location / {
        proxy_pass https://authentik;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade_keepalive;
    }
}

Application config:

server {
    # SSL and VHost configuration
    listen                  443 ssl http2;
    server_name             bans.domain2.tld;

    ssl_certificate         /etc/letsencrypt/live/subdomain.domain2.tld/fullchain.pem;
    ssl_certificate_key     /etc/letsencrypt/live/subdomain.domain2.tld/privkey.pem;

    # Increase buffer size for large headers
    # This is needed only if you get 'upstream sent too big header while reading response
    # header from upstream' error when trying to access an application protected by goauthentik
    proxy_buffers 8 16k;
    proxy_buffer_size 32k;

    location / {
        # Put your proxy_pass to your application here, and all the other statements you'll need
        proxy_pass          http://127.0.0.1:8888;
        proxy_set_header Host $host;

        ##############################
        # authentik-specific config
        ##############################
        auth_request     /outpost.goauthentik.io/auth/nginx;
        error_page       401 = @goauthentik_proxy_signin;
        auth_request_set $auth_cookie $upstream_http_set_cookie;
        add_header       Set-Cookie $auth_cookie;

        # translate headers from the outposts back to the actual upstream
        auth_request_set $authentik_username $upstream_http_x_authentik_username;
        auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
        auth_request_set $authentik_email $upstream_http_x_authentik_email;
        auth_request_set $authentik_name $upstream_http_x_authentik_name;
        auth_request_set $authentik_uid $upstream_http_x_authentik_uid;

        proxy_set_header X-authentik-username $authentik_username;
        proxy_set_header X-authentik-groups $authentik_groups;
        proxy_set_header X-authentik-email $authentik_email;
        proxy_set_header X-authentik-name $authentik_name;
        proxy_set_header X-authentik-uid $authentik_uid;
    }

    # all requests to /outpost.goauthentik.io must be accessible without authentication
    location /outpost.goauthentik.io {
        proxy_pass              https://auth.domain1.tld/outpost.goauthentik.io;
        # ensure the host of this vserver matches your external URL you've configured
        # in authentik
        proxy_set_header        Host $host;
        proxy_set_header        X-Original-URL $scheme://$http_host$request_uri;
        add_header              Set-Cookie $auth_cookie;
        auth_request_set        $auth_cookie $upstream_http_set_cookie;
        proxy_pass_request_body off;
        proxy_set_header        Content-Length "";
    }

    # Special location for when the /auth endpoint returns a 401,
    # redirect to the /start URL which initiates SSO
    location @goauthentik_proxy_signin {
        internal;
        add_header Set-Cookie $auth_cookie;
        return 302 /outpost.goauthentik.io/start?rd=$request_uri;
        # For domain level, use the below error_page to redirect to your authentik server with the full redirect path
        # return 302 https://auth.domain1.tld/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
    }
}

server {
    if ($host = bans.domain2.tld) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen         80;
    server_name    bans.domain2.tld;
    return 404; # managed by Certbot

}

server {
    server_name    internal-bans.domain2.tld;
    listen        127.0.0.1:8888;
    root /var/www/litebans;
    index index.php;

    access_log /var/log/nginx/litebans.app-access.log;
    error_log  /var/log/nginx/litebans.app-error.log error;

    # allow larger file uploads and longer script runtimes
    client_max_body_size 100m;
    client_body_timeout 120s;

    sendfile off;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param PHP_VALUE "upload_max_filesize = 100M \n post_max_size=100M";
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param HTTP_PROXY "";
        fastcgi_intercept_errors off;
        fastcgi_buffer_size 16k;
        fastcgi_buffers 4 16k;
        fastcgi_connect_timeout 300;
        fastcgi_send_timeout 300;
        fastcgi_read_timeout 300;
        include /etc/nginx/fastcgi_params;
    }

    location ~ /\.ht {
        deny all;
    }

Logs NGINX error log

2022/12/13 13:00:00 [error] 1371849#0: *40 no live upstreams while connecting to upstream, client: 162.158.238.213, server: bans.domain2.tld, request: "GET / HTTP/2.0", subrequest: "/outpost.goauthentik.io/auth/nginx", upstream: "https://auth.domain1.tld/outpost.goauthentik.io/auth/nginx", host: "bans.domain2.tld"
2022/12/13 13:00:00 [error] 1371849#0: *40 auth request unexpected status: 502 while sending to client, client: 162.158.238.213, server: bans.domain2.tld, request: "GET / HTTP/2.0", host: "bans.domain2.tld"
2022/12/13 13:00:01 [error] 1371849#0: *46 no live upstreams while connecting to upstream, client: 162.158.239.46, server: bans.domain2.tld, request: "GET /favicon.ico HTTP/2.0", subrequest: "/outpost.goauthentik.io/auth/nginx", upstream: "https://auth.domain1.tld/outpost.goauthentik.io/auth/nginx", host: "bans.domain2.tld", referrer: "https://bans.domain2.tld/"
2022/12/13 13:00:01 [error] 1371849#0: *46 auth request unexpected status: 502 while sending to client, client: 162.158.239.46, server: bans.domain2.tld, request: "GET /favicon.ico HTTP/2.0", host: "bans.domain2.tld", referrer: "https://bans.domain2.tld/"

There is nothing on Authentik logs about this (that I can see), but I am more than happy to provide more logs if needed.

[JantsoP]

So small(ish) update. I managed to get this somehow working. Originally the domains are protected by Cloudflare. But once I configured in NGINX to use IP instead of domain and added the cert given by authentik + disabled ssl verification on proxy_pass due self signed cert, it started working just fine. So could this be something possibly to do with Cloudflare?

EDIT: Forgot to mention that both domains use Cloudflare SSL certificates which I made in CF dashboard.

[adamzvolanek]

The procedure I have been needing to do is edit lines 41 and 59 each time to my servers IP address, similar to what you mentioned.

I too use CF and an CF SSL Cert. on the nginx side and none on the proxy_pass.

[AndrewBucklin]

I also ran into this same problem on 2023.3.0. I worked around it by hard-coding my Authentik server's IP address to bypass Cloudflare. If your NGINX can talk directly to Authentik, you could use the LAN IP. In my case, it could not, so I used the public WAN IP of my Authentik server.

Here are the relevant changes;

  location /outpost.goauthentik.io {
        proxy_pass              https://0.0.0.0/outpost.goauthentik.io;
        # ensure the host of this vserver matches your external URL you've configured
        # in authentik
        proxy_set_header        Host login.acme.com;
        ...

Possibly related: https://github.com/goauthentik/authentik/issues/4381 https://github.com/goauthentik/authentik/issues/3147

[eleith]

@BeryJu comment in https://github.com/goauthentik/authentik/issues/4496#issuecomment-1400898062 seems to relate to the root of this issue as well

in the instance web site on the providers page, on the setup section, my standalone nginx instructions renders with a FQDN host, so it set me down the wrong path initially.

this restriction does introduce a constraint against setting up authentik that only exposes its services behind a reverse proxy (therefore lacking access by IP only).

it's unclear what options you would have in those scenarios other than to punch a hole directly into authentik to guarantee it can be reached by IP as well as by FQDN.

are there any other workarounds to get FQDN working in proxy_pass ?

[adamzvolanek]

Unsure if this helps anyone setting NGINX up, however take a peek at my comment for a different issue for configuration?

https://github.com/goauthentik/authentik/issues/4381#issuecomment-1605195711

[wildwind123]

i just did it

 location /outpost.goauthentik.io {
              proxy_pass http://192.168.0.1:9003/outpost.goauthentik.io;
             ...
              proxy_redirect http://192.168.0.1:9003 https://auth.domen.com;

    }