Cannot login through caddy reverse proxy (403)

[nodeam]

Can the issue be reproduced with the latest available release? (y/n) yes

Which one is the environment gotify server is running in?

• [ ] Docker
• [*] Linux machine
• [ ] Windows machine
  Docker startup command or config file here (please mask sensitive information)

  
  

uname -a    
Linux pgotify 6.8.12-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-4 (2024-11-06T15:04Z) x86_64 GNU/Linux

lsb_release -a
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

It is a Proxmox LXC Container

cat /etc/systemd/system/gotify.service 
[Unit]
Description=Gotify
Requires=network.target
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/gotify
ExecStart=/opt/gotify/./gotify-linux-amd64
StandardOutput=append:/var/log/gotify/gotify.log
StandardError=append:/var/log/gotify/gotify-error.log
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

cat /opt/gotify/config.yml 
# Example configuration file for the server.
# Save it to `config.yml` when edited

server:
  keepaliveperiodseconds: 0 # 0 = use Go default (15s); -1 = disable keepalive; set the interval in which keepalive packets will be sent. Only change this value if you know what you are doing.
  listenaddr: "" # the address to bind on, leave empty to bind on all addresses. Prefix with "unix:" to create a unix socket. Example: "unix:/tmp/gotify.sock".
  port: 80 # the port the HTTP server will listen on

  ssl:
    enabled: true # if https should be enabled
    redirecttohttps: true # redirect to https if site is accessed by http
    listenaddr: "" # the address to bind on, leave empty to bind on all addresses. Prefix with "unix:" to create a unix socket. Example: "unix:/tmp/gotify.sock".
    port: 443 # the https port
    certfile: /etc/ssl/gotify/foo.bar.lan.crt # the cert file (leave empty when using letsencrypt)
    certkey: /etc/ssl/gotify/foo.bar.lan.key # the cert key (leave empty when using letsencrypt)
    letsencrypt:
      enabled: false # if the certificate should be requested from letsencrypt

  trustedproxies:
     - 127.0.0.1
     - 192.168.8.0/24
     - ::1

/etc/ssl/gotify/foo.bar.lan.crt generated by custom root-CA for the name and ip. Custom root-ca added to system certs.

ip a
...
inet 192.168.8.159/24 brd 192.168.8.255 scope global eth0
...

Do you have an reverse proxy installed in front of gotify server? (Please select None if the problem can be reproduced without the presense of a reverse proxy)

• [ ] None
• [ ] Nginx
• [ ] Apache
• [*] Caddy
  Reverse proxy configuration (please mask sensitive information)

  
  

cat /etc/caddy/Caddyfile 
{
        email foo@bar.tld
        crowdsec {
                api_key MfPxxxxxxxxxxxxxxxxxxxxxx
                api_url http://192.168.8.173:8080/
        }
}
import sites-enabled/*

cat /etc/caddy/sites-enabled/gotify-ssl-ip 
foo.bar.tld {
        reverse_proxy https://192.168.8.159:443 {
        header_up Host {upstream_hostport}
        header_up X-Real-IP {remote_host}
        }
        log {
        output file /var/log/caddy/foo.bar.tld-access.log
        }
}

On which client do you experience problems? (Select as many as you can see)

• [*] WebUI
• [ ] gotify-cli
• [ ] Android Client
• [ ] 3rd-party API call (Please include your code)

What did you do? Trying to login into WebUI on FQDN foo.bar.tld:443 getting "Login failed" and nothing in gotify logs while using any browser I own. Tested Firefox, Opera and Safari on debian, macos and android.

Loging into WebUI on local domain foo.bar.lan:443 or on 192.168.8.159:443 without any issue. Correct logins and simulated incorrect logins appear in /var/log/gotify/gotify.log. I never saw anything in /var/log/gotify/gotify-error.log

Is there anyway to set loglevel?

Loging in with Android Client works on both fqdn and local domain.

What did you expect to see? Successful login. I have identical caddy configurations for vaultwarden, nextcloudpi, photoprism and roundcube. All working without any problems.

What did you see instead? (Include screenshots, android logcat/request dumps if possible) Failed login

[eternal-flame-AD]

Thanks for the ticket.

It seems like you have a working WebUI but not a login and none of your requests hit gotify?

Can you look at the network and console tab in the devtools and upload what is intuitively not right?

Also a good idea might be to enable the access log on caddy and see how caddy route the request.

I will retag this as question for now because I think the fact that local access have logs written but the not working remote access has not which strongly suggests the requests did not even hit gotify.

[nodeam]

Caddy log for gotify

Hitting WebUI on fqdn:443

178.13.x.x - - [03/Dec/2024:14:33:55 +0000] "GET / HTTP/2.0" 200 1456
178.13.x.x  - - [03/Dec/2024:14:33:55 +0000] "GET /static/css/2.0f3898ba.chunk.css HTTP/2.0" 200 2467
178.13.x.x  - - [03/Dec/2024:14:33:55 +0000] "GET /static/js/main.ca03a2c1.chunk.js HTTP/2.0" 200 16490
178.13.x.x  - - [03/Dec/2024:14:33:55 +0000] "GET /static/js/2.64bfa034.chunk.js HTTP/2.0" 200 260550
178.13.x.x  - - [03/Dec/2024:14:33:56 +0000] "GET /static/favicon-32x32.png HTTP/2.0" 200 1942
178.13.x.x  - - [03/Dec/2024:14:33:56 +0000] "GET /static/favicon-196x196.png HTTP/2.0" 200 15750

Entering credentials

178.13.x.x - - [03/Dec/2024:14:34:24 +0000] "POST /client HTTP/2.0" 403 0

Edit: Console tab in the devtools says nothing especially because using the same browser instance on:

• local domain:433 - login is ok
• fqdn:443 - I get 403

[eternal-flame-AD]

This is highly unlikely to be a genuine gotify response because the only place 403 is returned is if you used an application token for a client operation or vice versa, and you should see a JSON error message indicating the error on the browser devtools if you look at the corresponding request, I assume the 0 at the end mean content-length 0 but a genuine error message should have a non zero content-length.

If it helps I use Caddy too and here is my config and it just works, can you try simplify your setup (like use HTTP for the caddy <-> gotify part) and gradually try add back the additional options?

https://gotify.yumechi.jp {
    reverse_proxy :4101
    import /etc/caddy/include.d/tls // just TLS config
}

[eternal-flame-AD]

Also just to not miss the obvious have you tried looking at the log of Caddy itself?

[jmattheis]

FYI: Gotify does return 403 with empty response when there is a failed server side cors request.