search

lucaslorentz / caddy-docker-proxy

Caddy as a reverse proxy for Docker
MIT License
2.61k stars 163 forks source link

Caddy Docker Proxy + Authelia + (Cloudflare Argo Tunnel) #573

Closed ameeno closed 5 months ago

ameeno commented 5 months ago

Hi everyone,

First off, massive thanks for this project, I have Caddy docker proxy + Authelia setup on my docker host – it's running brilliantly (with sablier)!

Currently, I'm using a seperate cloudflare-ddns container for updating my domain root's 'A' record on Cloudflare whenever my IP changes. Plus, I've got ports 80 and 443 open and forwarded to the docker host IP on my home router. It's all working fine, but there are two security concerns: 1) My home IP is publicly visible, and 2) I need to open ports in my home network.

I attempted to set up a Cloudflare Argo Tunnel in front of Caddy, but unfortunately, it's not functioning, and I'm stumped as to why.

Here's my Dockerfile for the Caddy container:

# syntax=docker/dockerfile:labs
ARG CADDY_VERSION=2.7
FROM caddy:${CADDY_VERSION}-builder AS builder

ADD https://github.com/acouvreur/sablier.git#beta /sablier
RUN xcaddy build \
    --with github.com/lucaslorentz/caddy-docker-proxy/v2 \
    --with github.com/acouvreur/sablier/plugins/caddy=/sablier/plugins/caddy \
    --with github.com/caddy-dns/cloudflare 

FROM caddy:${CADDY_VERSION}-alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

CMD ["caddy", "docker-proxy"]

And here's a part of the Caddyfile generated by the container:

{
        acme_dns cloudflare <CLOUDFLARE-API-TOKEN>
        email <CLOUDFLARE-EMAIL-ADDRESS>
        order sablier before reverse_proxy
}
(authelia) {
        forward_auth authelia:9091 {
                copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
                import trusted_proxy_list
                uri /api/verify?rd=https://auth.domain.tld
        }
}
(sablier) {
        import authelia
        sablier {
                dynamic
                group {args[0]}
        }
}
(trusted_proxy_list) {
        trusted_proxies 10.0.0.0/8 172.16.0.0/16 192.168.0.0/16 fc00::/7
}
auth.domain.tld {
        reverse_proxy authelia:9091 {
                import trusted_proxy_list
        }
}
bookstack.domain.tld {
        import authelia
        reverse_proxy bookstack:80 {
                import trusted_proxy_list
        }
}
calibre.domain.tld {
        import sablier calibre
        reverse_proxy calibre-web:8083 {
                import trusted_proxy_list
        }
}
qb.domain.tld {
        import authelia
        reverse_proxy 192.168.0.111:3184 {
                import trusted_proxy_list
        }
}

I've gone through several guides, including this one on using Caddy with Cloudflared Argo Tunnel and DNS wildcard, and another on setting up the Caddy and Cloudflare network.

My goal is to route *.domain.tld through the Cloudflare tunnel to the docker host and then to Caddy, just as efficiently as it's running with my current router setup.

I'm at a loss here – any help or suggestions would be greatly appreciated! Thanks in advance!

francislavoie commented 5 months ago

You can't just say "it's not working". That gives us no clues. Show evidence of the problem. Show an example request with curl -v. Show your Caddy logs. Explain the actual behaviour you're seeing.

ameeno commented 5 months ago

You can't just say "it's not working". That gives us no clues. Show evidence of the problem. Show an example request with curl -v. Show your Caddy logs. Explain the actual behaviour you're seeing.

Right, That's correct, I was hoping someone who had tried it before could mention some pitfalls I should know about.

Well, I have set up the tunnel and updated cloudflare and local dns resolver and tried to access a service, here is what i'm getting:

nslookup service.domain.tld
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
service.domain.tld  canonical name = TUNNEL_ID.cfargotunnel.com.
Name:   TUNNEL_ID.cfargotunnel.com
Address: fd10:aec2:5dae::

✦ bash ~ ❯ traceroute fd10:aec2:5dae::
traceroute to fd10:aec2:5dae:: (fd10:aec2:5dae::), 30 hops max, 80 byte packets
 1  2a02:6b62:c94::1 (2a02:6b62:c94::1)  3.525 ms  3.409 ms  3.360 ms
 2  2a02:6b60:0:5::1 (2a02:6b60:0:5::1)  4.845 ms !X  4.797 ms !X  4.745 ms !X

✦ bash ~ ❯ curl -v service.domain.tld
* processing: service.domain.tld
*   Trying [fd10:aec2:5dae::]:80...
* connect to fd10:aec2:5dae:: port 80 failed: Permission denied
* Failed to connect to service.domain.tld port 80 after 34 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to service.domain.tld port 80 after 34 ms: Couldn't connect to server

on the docker cloudflared container logs I get nothing and see no traffic, I have checked my dns resolvers for ipv4 and ipv6.

Honestly I would prefer to use IPv4 only, but not sure this is possible with an argo tunnel.

The guides all say to put a cname record for your url (or wildcard) to the TUNNEL_ID.cfargotunnel.com right here: https://community.cloudflare.com/t/wildcard-subdomains/501612

ameeno commented 5 months ago

OK did a bit more tinkering with cloudflare, and managed to get the tunnel working, but now the problem seems to be between cloudflared & caddy.

Logs i get are:

2024-01-19T22:04:52Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" connIndex=3 dest=https://homepage.domain.tld/ ip=198.41.192.27 type=http
2024-01-19T22:04:52Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" cfRay=8482603bd91e6394-LHR ingressRule=2 originService=https://caddy:443
2024-01-19T22:04:52Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" connIndex=3 dest=https://homepage.domain.tld/favicon.ico ip=198.41.192.27 type=http
ameeno commented 5 months ago

Perhaps some more info is here:

I curled a service that is reaching the cloudflared tunnel and the host, but dies at caddy:

 bash ~ ❯ curl -v https://service.domain.tld
* processing: https://service.domain.tld
*   Trying [2606:4700:3034::ac43:ac32]:443...
*   Trying 172.67.172.50:443...
* Connected to service.domain.tld (172.67.172.50) port 443
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=domain.tld
*  start date: Jan 18 00:20:51 2024 GMT
*  expire date: Apr 17 00:20:50 2024 GMT
*  subjectAltName: host "service.domain.tld" matched cert's "*.domain.tld"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: service.domain.tld]
* h2 [:path: /]
* h2 [user-agent: curl/8.2.1]
* h2 [accept: */*]
* Using Stream ID: 1
> GET / HTTP/2
> Host: service.domain.tld
> User-Agent: curl/8.2.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 502 
< date: Fri, 19 Jan 2024 22:28:16 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 15
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=7lDlfMLf7A6AHl%2FkOZ4d%2FB4ORJIpY48lagnYTDkcSvJ8iZWoO2TrdksWjzNi7JvaAr2Rn4T0mMG5srtXOvKCT20k0XfI35XOhSS3%2FSKC%2Fvdl4NkYpDe8GQYGNv4T7vc5aw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< x-frame-options: SAMEORIGIN
< referrer-policy: same-origin
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< server: cloudflare
< cf-ray: 8482828478367324-LHR
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host service.domain.tld left intact
error code: 502

Seems like i'm getting a 502 error and it cant reach the caddy service, but after getting into the cloudflared container, I can ping caddy service fine.

Really confused about it :(

ameeno commented 5 months ago

Hi all,

Just to let you know, I did some further debugging, Involving putting an nginx port 80 http container up and a new ingress going straight to that container.

The tunnel worked fine after some DNS mapping on cloudflare. This enabled me to figure out that there was a problem with certificate generation between cloudflare tunnel and caddy.

I added the following to my caddy file and it all started working fine!!

      caddy_99: "*.domain.tld, domain.tld"
      caddy_99.tls.dns: "cloudflare {env.CLOUDFLARE_API_TOKEN}"

Prior to this all my caddy entries were of the type .domain.tld, and it appeared caddy was not doing dns validation only zerossl and so origins on the certificates were not matching up :)

Thanks, problem solved. leaving this up incase anyone runs into the same issues.