search

opnsense / plugins

OPNsense plugin collection
https://opnsense.org/
BSD 2-Clause "Simplified" License
810 stars 593 forks source link

Caddy: blank pages on all subdomains under wildcard domain #4024

Closed no-usernames-left closed 3 weeks ago

no-usernames-left commented 3 weeks ago

Important notices

Describe the bug All subdomains being served by Caddy return a blank page (or an aborted connection when the applicable box is checked).

Extract from OPNsense configuration XML

  <Pischem>
    <caddy version="1.1.8">
      <general>
        <enabled>1</enabled>
        <TlsEmail>REDACTED</TlsEmail>
        <TlsAutoHttps/>
        <TlsDnsProvider>gandi</TlsDnsProvider>
        <TlsDnsApiKey>REDACTED</TlsDnsApiKey>
        <TlsDnsSecretApiKey/>
        <TlsDnsOptionalField1/>
        <TlsDnsOptionalField2/>
        <TlsDnsOptionalField3/>
        <TlsDnsOptionalField4/>
        <accesslist/>
        <abort>0</abort>
        <LogCredentials>0</LogCredentials>
        <LogAccessPlain>0</LogAccessPlain>
        <LogAccessPlainKeep>10</LogAccessPlainKeep>
        <DynDnsSimpleHttp/>
        <DynDnsInterface>opt1</DynDnsInterface>
        <DynDnsCheckInterval>1</DynDnsCheckInterval>
        <DynDnsIpVersions>ipv4</DynDnsIpVersions>
        <DynDnsTTL>1</DynDnsTTL>
      </general>
      <reverseproxy>
        <reverse uuid="906646cb-2535-4a3e-b71d-fca6070dea3f">
          <enabled>1</enabled>
          <FromDomain>*.DOMAIN.TLD</FromDomain>
          <FromPort>443</FromPort>
          <accesslist/>
          <basicauth/>
          <description>*.DOMAIN.TLD:443</description>
          <DnsChallenge>1</DnsChallenge>
          <CustomCertificate/>
          <AccessLog>1</AccessLog>
          <DynDns>0</DynDns>
          <AcmePassthrough/>
        </reverse>
        <subdomain uuid="b6d464c7-4e4f-4b1c-9f0b-3a2d2e04cb54">
          <enabled>1</enabled>
          <reverse>906646cb-2535-4a3e-b71d-fca6070dea3f</reverse>
          <FromDomain>fw.DOMAIN.TLD</FromDomain>
          <FromPort>443</FromPort>
          <accesslist/>
          <basicauth/>
          <description>OPNsense Web GUI</description>
          <DynDns>1</DynDns>
        </subdomain>
        <subdomain uuid="df5129a9-caa6-493a-a2ca-60406901ddbb">
          <enabled>1</enabled>
          <reverse>906646cb-2535-4a3e-b71d-fca6070dea3f</reverse>
          <FromDomain>ds.DOMAIN.TLD</FromDomain>
          <FromPort>443</FromPort>
          <accesslist/>
          <basicauth/>
          <description>DiskStation Web GUI</description>
          <DynDns>0</DynDns>
        </subdomain>
        <handle uuid="76df0f2b-387a-4219-9799-942ad94e5c69">
          <enabled>1</enabled>
          <reverse>906646cb-2535-4a3e-b71d-fca6070dea3f</reverse>
          <subdomain>b6d464c7-4e4f-4b1c-9f0b-3a2d2e04cb54</subdomain>
          <HandleType>handle</HandleType>
          <HandlePath/>
          <header/>
          <ToDomain>127.0.0.1</ToDomain>
          <ToPort>8443</ToPort>
          <ToPath/>
          <PassiveHealthFailDuration/>
          <HttpTls>1</HttpTls>
          <HttpNtlm>0</HttpNtlm>
          <HttpTlsInsecureSkipVerify>1</HttpTlsInsecureSkipVerify>
          <HttpTlsTrustedCaCerts/>
          <HttpTlsServerName/>
          <description>OPNsense Web GUI</description>
        </handle>
        <handle uuid="611ec12e-4d08-4d0a-9eef-237667a60c02">
          <enabled>1</enabled>
          <reverse>906646cb-2535-4a3e-b71d-fca6070dea3f</reverse>
          <subdomain>df5129a9-caa6-493a-a2ca-60406901ddbb</subdomain>
          <HandleType>handle</HandleType>
          <HandlePath/>
          <header/>
          <ToDomain>192.168.X.Y</ToDomain>
          <ToPort>NNNN</ToPort>
          <ToPath/>
          <PassiveHealthFailDuration/>
          <HttpTls>0</HttpTls>
          <HttpNtlm>0</HttpNtlm>
          <HttpTlsInsecureSkipVerify>0</HttpTlsInsecureSkipVerify>
          <HttpTlsTrustedCaCerts/>
          <HttpTlsServerName/>
          <description>DiskStation Web GUI</description>
        </handle>
      </reverseproxy>
    </caddy>
  </Pischem>

Caddyfile

# DO NOT EDIT THIS FILE -- OPNsense auto-generated file

# Global Options
{
        log {
                include http.log.access.906646cb-2535-4a3e-b71d-fca6070dea3f
                output net unixgram//var/caddy/var/run/log {
                }
                format json {
                        time_format rfc3339
                }
        }

        dynamic_dns {
                provider gandi REDACTED
                domains {
                        DOMAIN.TLD fw
                }
                ip_source interface igc2
                check_interval 1m
                versions ipv4
                ttl 1h
        }

        email REDACTED
        import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration

# Reverse Proxy Domain: "906646cb-2535-4a3e-b71d-fca6070dea3f"
*.DOMAIN.TLD:443 {
        log 906646cb-2535-4a3e-b71d-fca6070dea3f
        tls {
                dns gandi REDACTED
        }

        @b6d464c7-4e4f-4b1c-9f0b-3a2d2e04cb54 {
                host fw.DOMAIN.TLD:443
        }
        handle @b6d464c7-4e4f-4b1c-9f0b-3a2d2e04cb54 {
                handle {
                        reverse_proxy 127.0.0.1:8443 {
                                transport http {
                                        tls_insecure_skip_verify
                                }
                        }
                }
        }
        @df5129a9-caa6-493a-a2ca-60406901ddbb {
                host ds.DOMAIN.TLD:443
        }
        handle @df5129a9-caa6-493a-a2ca-60406901ddbb {
                handle {
                        reverse_proxy 192.168.X.Y:NNNN {
                        }
                }
        }
}

import /usr/local/etc/caddy/caddy.d/*.conf

caddy validate output

2024/06/04 02:45:56.169 INFO    using adjacent Caddyfile
2024/06/04 02:45:56.169 INFO    using provided configuration    {"config_file": "Caddyfile", "config_adapter": ""}
2024/06/04 02:45:56.169 WARN    No files matching import glob pattern   {"pattern": "/usr/local/etc/caddy/caddy.d/*.global"}
2024/06/04 02:45:56.170 WARN    No files matching import glob pattern   {"pattern": "/usr/local/etc/caddy/caddy.d/*.conf"}
2024/06/04 02:45:56.172 INFO    redirected default logger       {"from": "stderr", "to": "unixgram//var/caddy/var/run/log"}
Valid configuration

Relevant log files Starting Caddy plugin, waiting a few seconds, then trying to access https://fw.DOMAIN.TLD/:

"info","ts":"2024-06-04T02:57:02Z","logger":"admin","msg":"admin endpoint started","address":"unix//var/run/caddy/caddy.sock","enforce_origin":false,"origins":["","//127.0.0.1","//::1"]}

"info","ts":"2024-06-04T02:57:02Z","msg":"autosaved config (load with --resume flag)","file":"/var/db/caddy/config/caddy/autosave.json"}

"info","ts":"2024-06-04T02:57:02Z","msg":"serving initial configuration"}

"info","ts":"2024-06-04T02:57:19Z","logger":"http.log.access.906646cb-2535-4a3e-b71d-fca6070dea3f","msg":"handled request","request":{"remote_ip":"REDACTED","remote_port":"45052","client_ip":"REDACTED","proto":"HTTP/2.0","method":"GET","host":"fw.DOMAIN.TLD","uri":"/","headers":{"Accept-Encoding":["gzip, deflate, br, zstd"],"Te":["trailers"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Sec-Fetch-Mode":["navigate"],"Sec-Gpc":["1"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Priority":["u=1"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"],"Dnt":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"fw.DOMAIN.TLD"}},"bytes_read":0,"user_id":"","duration":0.000013233,"size":0,"status":0,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}

Additional context Followed the how-to essentially to the letter, but using a wildcard domain and DNS-01 challenges instead, and not adding an inbound allow rule for 80/tcp in the firewall to prevent any non-TLS inbound connections.

Redactions are all upper-case and should be obvious.

Environment OPNsense Business 24.4_8 Caddy 1.5.4_1

Monviech commented 3 weeks ago

Hey there,

your configuration looks fine and valid, though could be improved.

I would first try to make all domain and subdomain ports empty. If that solves the issue I have a good indicator what goes wrong.

For the blank page issue, that means that the routing to the upstream is not working, maybe the upstream can't be accessed. Does it also not work for 127.0.0.1:8443 since that would surprise me.

For further troubleshooting, it would be best if you explain your case in https://caddy.community . They know that this plugin exists and will help you.

Monviech commented 3 weeks ago

Yeah I have confirmed that subdomains are not allowed to have ports. That is a mistake in the GUI. I will have to fix that.

Subdomains inherit the port(s) of their wildcard *.domain

no-usernames-left commented 3 weeks ago

Hello,

Thanks for taking the time to look at my issue.

  • Caddy automatically upgrades connections from HTTP to HTTPS, there won't ever be insecure connections.

Unfortunately the Secure flag is not set on cookies as often as I'd prefer, and if they are able to connect on port 80, any non-Secure cookie already set on the client will be sent in the clear before Caddy has a chance to redirect them to HTTPS. This results in the loss of control of, as one example, the session token.

Therefore, port 80 will be remaining firewalled; you may wish to revise this recommendation.

Yeah I have confirmed that subdomains are not allowed to have ports. That is a mistake in the GUI. I will have to fix that.

Good to know that I'm actually not going crazy.

After removing the ports from both subdomains and applying the changes, ds.DOMAIN.TLD works, but fw.DOMAIN.TLD does not:

"error","ts":"2024-06-04T18:21:43Z","logger":"http.log.access.906646cb-2535-4a3e-b71d-fca6070dea3f","msg":"handled request","request":{"remote_ip":"REDACTED","remote_port":"34706","client_ip":"REDACTED","proto":"HTTP/2.0","method":"GET","host":"fw.DOMAIN.TLD","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Dnt":["1"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"],"Te":["trailers"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Priority":["u=1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"fw.DOMAIN.TLD"}},"bytes_read":0,"user_id":"","duration":5.031218497,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
Monviech commented 3 weeks ago

I will hint the cookie issue in the docs, thank you for clarifying.

Good that one of your domains works now.

The other shows "502 Bad Gateway" which means the response was unexpected. I guess you try to reverse proxy the OPNsense WebGUI. Your configuration looks the same as my configuration, so I don't know what the issue could be right now.

Maybe debug logs would help, but in the version of os-caddy in the OPNsense BE edition they can't be enabled yet from the GUI.

To compare, here is my personal Caddyfile configuration that works for my OPNsense WebGUI, the only difference is that I'm not using subdomains (though it should not matter) and have appended an access list:

# Reverse Proxy Domain: "e40998d9-94cf-4ca5-85df-1d7479acf375"
fw.example.com {
    log {
        output file /var/log/caddy/access/e40998d9-94cf-4ca5-85df-1d7479acf375.log {
            roll_keep_for 2d
        }
    }

    @86600cf6-e1d0-46d4-9836-d872a9c4e0e1 {
        client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
    }

    handle @86600cf6-e1d0-46d4-9836-d872a9c4e0e1 {
        handle {
            reverse_proxy 127.0.0.1:4444 {
                transport http {
                    tls_insecure_skip_verify
                }
            }
        }
    }

    abort
}

Maybe a total restart of caddy would help to totally flush any old config. In later versions there have been changes that "force reloads" caddy, which will always regenerate the whole config.

Monviech commented 3 weeks ago

@no-usernames-left I think I have an idea. Did you bind your OPNsense WebGUI to specific interfaces? Because then the bind for it on 127.0.0.1 does not exist, since it's not bound to the wildcard interface. You have to use the IP address of the interface where the WebGUI has been bound to.

no-usernames-left commented 3 weeks ago

Did you bind your OPNsense WebGUI to specific interfaces?

No, Listen Interfaces is set to All (recommended).

Monviech commented 3 weeks ago

@no-usernames-left

Okay thank you for the information. Then I have no idea right now.

no-usernames-left commented 3 weeks ago

Unfortunately after uninstalling the plugin, manually removing Caddyfile and all the Pischem section of the OPNsense configuration file, rebooting, reinstalling Caddy, and configuring everything from scratch, I'm still getting an error when trying to access the OPNsense GUI via Caddy:

"error","ts":"2024-06-07T14:53:30Z","logger":"http.log.error","msg":"read tcp 127.0.0.1:18822->127.0.0.1:8443: read: connection reset by peer","request":{"remote_ip":"REDACTED","remote_port":"43182","client_ip":"REDACTED","proto":"HTTP/2.0","method":"GET","host":"fw.DOMAIN.TLD","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Dnt":["1"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Site":["cross-site"],"Priority":["u=1"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"fw.DOMAIN.TLD"}},"duration":20.09877844,"status":502,"err_id":"4ui2pzaf2","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

So I SSH'd into OPNsense and went digging, and it looks like I figured out the cause of the issue:

# sockstat | grep 8443
root     lighttpd   12050 7  tcp4   *:8443                *:*
root     lighttpd   12050 8  tcp6   *:8443                *:*

# openssl s_client -connect 127.0.0.1:8443 -servername fw.DOMAIN.TLD
CONNECTED(00000003)
write:errno=54
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

# openssl s_client -connect 127.0.0.1:8443
CONNECTED(00000003)
write:errno=54
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 293 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
#

But yet I'm totally able to reach the device on port 8443 externally, which is how I'm configuring Caddy in the first place.

This looks like an upstream issue; @fichtner can you confirm if localhost is being handled differently re the certificate selection? (The default OPNsense self-signed certificate is in place at an OPNsense level and that's it.)

fichtner commented 3 weeks ago

I have this lighttpd version at the moment:

# pkg info | grep lighttpd
lighttpd-1.4.76                Secure, fast, compliant, and flexible Web Server

And when probing localhost (on standard 443) I'm getting certificates with s_client for both cases you've posted. Same as with external IP. Not sure what's going on with your install.

Cheers, Franco

no-usernames-left commented 3 weeks ago

I have this lighttpd version at the moment:

# pkg info | grep lighttpd
lighttpd-1.4.76                Secure, fast, compliant, and flexible Web Server

And when probing localhost (on standard 443) I'm getting certificates with s_client for both cases you've posted. Same as with external IP. Not sure what's going on with your install.

Cheers, Franco

This box has 1.4.75, claims to be up to date (OPNsense Business 24.4_8) and is more or less a virgin install with just a couple of firewall rules, one LAN, one WAN, and one Wireguard server interface. What output would help you troubleshoot?

fichtner commented 3 weeks ago

on a stock 24.4 I can't reproduce. certificates show on localhost, but I'm not sure if that was your issue or I missed the point.

no-usernames-left commented 3 weeks ago

Did you try when the GUI port is set to 8443?

Where can I look on the filesystem or in the UI to fix the issue? This box is in prod so I can't just wipe it and reinstall.

Monviech commented 3 weeks ago

As reference this is the configuration that is not working.

https://docs.opnsense.org/manual/how-tos/caddy.html#reverse-proxy-the-opnsense-webui

I have it working in my OPNsense, also with certificate skipping. And I have talked to multiple users in the past who also use it like this (with proper cert handling like in the docs), or certificate skipping (tls_insecure_skip_verify)

So Im confident that it should work when reproducing it.

no-usernames-left commented 3 weeks ago

Indeed, I have Caddy set to ignore the fact that the presented certificate will be the default self-signed one, but the problem is that lighttpd is not presenting any certificate and so Caddy cannot proceed.

In other words, what can I poke to restore lighttpd's default config (ideally other than listening port)?

no-usernames-left commented 3 weeks ago

Okay, so I'm actually seeing the same behaviour as here when I try connecting to any combination of [localhost,WAN] and [443,8443] from a root shell.

First I get CONNECTED(00000003), then a good solid wait, then write:errno=54 (ECONNRESET) and the rest of the output.

Is this some kind of strange ruleset issue which is allowing the three-way handshake but then dropping all traffic on the floor?

no-usernames-left commented 3 weeks ago

I found the cause of the issue.

In Firewall - Settings - Advanced, I changed Enable syncookies from always back to never (default) and the issue was immediately resolved.

@fichtner It would appear as though, in this configuration, enabling syncookies breaks traffic coming from the firewall itself.