breakfix: multiple Caddy servers - redirect loop

[AlexDaichendt]

Describe the issue

Hey! I have two hosts in different countries each with their own caddy server. Host A (a.example.com) hosts a bunch of services including an auth portal, which secures certain endpoints with ldap. I followed the instructions and it worked.

Now I added Host B (b.example.com) to the network. Since I thought I might get to reuse the auth portal of Host a, I only added the authorization policy to Host B leaving out the identity store and authentication portal (full config below).

I am able to browse an endpoint on b.example.com and get redirected to the auth portal, can login, but then the redirect back to the original application fails with a Too many redirects error. The console reveals, that caddy on B - after being redirected from the auth portal - still redirects back to the auth portal causing a loop. The access cookie is however set. If I reopen the original page I can now browse it, so the authentication was indeed successful.

Maybe something is wrong, I can't quite figure it out tho. I would very much appreciate a pointer.

Configuration

Host A:

{
    servers {
        metrics
    }
    order authenticate before respond
    order authorize before basicauth

    security {

        ldap identity store example.com {
            realm example.com
            servers {
                ldap://lldap:3890
            }
            attributes {
                name displayName
                                surename cn
                username uid
                member_of memberOf
                email mail
            }
            username "CN=admin,OU=people,DC=example,DC=com"
            password {env.LDAP_PASSWD}
            search_base_dn "DC=example,DC=com"
            search_filter "(&(uid=%s)(objectClass=person))"
            groups {
                "uid=user,ou=groups,dc=example,dc=com" authp/user
            }
        }

        authentication portal myportal {
            crypto default token lifetime 3600
            crypto key sign-verify {env.JWT_SHARED_KEY}
            enable identity store example.com
            cookie domain example.com
            ui {
                logo url "https://caddyserver.com/resources/images/caddy-circle-lock.svg"
                logo description "Caddy"
                links {
                    "My Identity" "/whoami" icon "las la-user"
                }
                #password_recovery_enabled yes
            }
        }

        authorization policy mypolicy {
            # disable auth redirect
            set auth url https://auth.example.com

            crypto key verify {env.JWT_SHARED_KEY}
                allow roles authp/user
                }
    }
}
(tls) {
    tls {
        dns cloudflare {env.CF_API_KEY}
    }
}

auth.example.com {
    route {
        authenticate with myportal
    }
}

example.com {
  root * /config/html
  authorize with mypolicy
  encode gzip
  file_server browse
  import tls
}

Host B:

{
    servers {
        metrics
    }

    order authenticate before respond
    order authorize before basicauth

    security {

      authorization policy mypolicy {
        # disable auth redirect
        set auth url https://auth.example.com
        crypto key verify {env.JWT_SHARED_KEY}
        allow roles authp/user

      }
    }
}

(tls) {
    tls {
        dns cloudflare {env.CF_API_KEY}
    }
}

:2020 {
  metrics /metrics
}

status.example.com {
  authorize with mypolicy
  reverse_proxy http://statping:8080
  import tls
}
~

Version Information

Provide output of caddy list-modules -versions | grep git below: apparently it is empty.

Expected behavior

The original application should not redirect back to the auth portal, I suppose

[greenpau]

@AlexDaichendt , please login to the portal and share what you see in /whoami.

[AlexDaichendt]

Absolutely!

{
  "addr": "10.10.10.252",
  "authenticated": true,
  "email": "alex@---------",
  "exp": 1664380296,
  "expires_at_utc": "Wed Sep 28 15:51:36 UTC 2022",
  "iat": 1664376696,
  "iss": "https://auth.example.com/login",
  "issued_at_utc": "Wed Sep 28 14:51:36 UTC 2022",
  "jti": "CUiHmKNr2Yn6XVxMx36QYWLNQDXaanuKu3ThV",
  "name": "Alex D",
  "nbf": 1664376636,
  "not_before_utc": "Wed Sep 28 14:50:36 UTC 2022",
  "origin": "example.com",
  "roles": [
    "authp/user"
  ],
  "sub": "alex"
}

[greenpau]

Since I thought I might get to reuse the auth portal of Host a, I only added the authorization policy to Host B leaving out the identity store and authentication portal (full config below).

@AlexDaichendt , this part looks correct.

The console reveals, that caddy on B - after being redirected from the auth portal - still redirects back to the auth portal causing a loop.

What does the log say on B? Please enabled debug so it tells you the reason for the redirect.

Also, please check that the JWT_SHARED_KEY matches.

[AlexDaichendt]

The JWT_SHARED_KEY matches.

Debug log (the last 4 messages repeat for about 20 times in total):

{"level":"debug","ts":1664377389.4035413,"logger":"events","msg":"event","name":"tls_get_certificate","id":"b57b4da7-c682-485a-912e-a5f5c4256af2","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867],"ServerName":"status.example.com","SupportedCurves":[29,23,24],"SupportedPoints":null,"SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513],"SupportedProtos":["h3"],"SupportedVersions":[772],"Conn":{}}}}
{"level":"debug","ts":1664377389.4037406,"logger":"tls.handshake","msg":"choosing certificate","identifier":"status.example.com","num_choices":1}
{"level":"debug","ts":1664377389.4038074,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"status.example.com","subjects":["status.example.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"e95ddd9c946ba2d8e802401a122e4cbab1ded0e7a2ebc6044ff684ca0e022053"}
{"level":"debug","ts":1664377389.4039533,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"REDACTED","remote_port":"38764","subjects":["status.example.com"],"managed":true,"expiration":1672139201,"hash":"e95ddd9c946ba2d8e802401a122e4cbab1ded0e7a2ebc6044ff684ca0e022053"}
{"level":"debug","ts":1664377389.431988,"logger":"security","msg":"token validation error","session_id":"bTZqbeIx6EMRZFk4pUEgRo0b4YsWdtvHHAu5","request_id":"5b9c388e-3f5d-408f-be11-bef3f3955315","error":"no token found"}
{"level":"debug","ts":1664377389.4321094,"logger":"security","msg":"redirecting unauthorized user","session_id":"bTZqbeIx6EMRZFk4pUEgRo0b4YsWdtvHHAu5","request_id":"5b9c388e-3f5d-408f-be11-bef3f3955315","method":"location"}
{"level":"error","ts":1664377389.4322612,"logger":"http.handlers.authentication","msg":"auth provider returned error","provider":"authorizer","error":"user authorization failed: src_ip=REDACTED, src_conn_ip=REDACTED, reason: no token found"}
{"level":"debug","ts":1664377389.432415,"logger":"http.log.error","msg":"not authenticated","request":{"remote_ip":"REDACTED","remote_port":"38764","proto":"HTTP/3.0","method":"GET","host":"status.example.com","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"Cookie":[],"Sec-Ch-Ua":["\"Chromium\";v=\"105\", \"Not)A;Brand\";v=\"8\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Linux\""],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","server_name":""}},"duration":0.000485005,"status":401,"err_id":"4qt7bfhen","err_trace":"caddyauth.Authentication.ServeHTTP (caddyauth.go:88)"}
{"level":"debug","ts":1664377394.3181589,"logger":"security","msg":"token validation error","session_id":"bTZqbeIx6EMRZFk4pUEgRo0b4YsWdtvHHAu5","request_id":"80e91424-b096-451b-9fcf-e199789c0f82","error":"keystore: failed to parse token"}
{"level":"debug","ts":1664377394.318194,"logger":"security","msg":"redirecting unauthorized user","session_id":"bTZqbeIx6EMRZFk4pUEgRo0b4YsWdtvHHAu5","request_id":"80e91424-b096-451b-9fcf-e199789c0f82","method":"location"}
{"level":"error","ts":1664377394.3182328,"logger":"http.handlers.authentication","msg":"auth provider returned error","provider":"authorizer","error":"user authorization failed: src_ip=REDACTED, src_conn_ip=REDACTED, reason: keystore: failed to parse token"}
{"level":"debug","ts":1664377394.3182683,"logger":"http.log.error","msg":"not authenticated","request":{"remote_ip":"REDACTED","remote_port":"38764","proto":"HTTP/3.0","method":"GET","host":"status.example.com","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Referer":["https://auth.example.com/"],"Cookie":[],"Cache-Control":["max-age=0"],"Sec-Fetch-Site":["same-site"],"Sec-Ch-Ua":["\"Chromium\";v=\"105\", \"Not)A;Brand\";v=\"8\""],"Sec-Ch-Ua-Platform":["\"Linux\""]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","server_name":""}},"duration":0.000234483,"status":401,"err_id":"kkt610cm9","err_trace":"caddyauth.Authentication.ServeHTTP (caddyauth.go:88)"}
{"level":"debug","ts":1664377394.543872,"logger":"security","msg":"token validation error","session_id":"bTZqbeIx6EMRZFk4pUEgRo0b4YsWdtvHHAu5","request_id":"2935e56d-948f-4ff1-ab9b-2f0d2b9ff170","error":"keystore: failed to parse token"}
{"level":"debug","ts":1664377394.5439,"logger":"security","msg":"redirecting unauthorized user","session_id":"bTZqbeIx6EMRZFk4pUEgRo0b4YsWdtvHHAu5","request_id":"2935e56d-948f-4ff1-ab9b-2f0d2b9ff170","method":"location"}
{"level":"error","ts":1664377394.5439456,"logger":"http.handlers.authentication","msg":"auth provider returned error","provider":"authorizer","error":"user authorization failed: src_ip=REDACTED, src_conn_ip=REDACTED, reason: keystore: failed to parse token"}

[greenpau]

@AlexDaichendt , "reason: no token found" means the cookie was not visible to Site B. Please login to /whoami, inspect your cookies (Chrome Dev Tools) and let me know what domain it is associated with.

[greenpau]

Please provide output of:

caddy list-modules -versions | egrep "(auth|security)"

[greenpau]

@AlexDaichendt , for example here is my domain from github cookies.

[AlexDaichendt]

In the dev tools for the two cookies access and AUTHP_SESSION_ID it says domain ".example.com"

/srv # caddy list-modules --versions | egrep "(auth|security)"
http.authentication.hashes.bcrypt v2.6.1
http.authentication.hashes.scrypt v2.6.1
http.authentication.providers.http_basic v2.6.1
http.handlers.authentication v2.6.1
tls.client_auth.leaf v2.6.1
http.authentication.providers.authorizer v1.1.15
http.handlers.authenticator v1.1.15
security v1.1.15

[greenpau]

@AlexDaichendt , also, try adding the following after cookie domain example.com.

cookie domain example.com
cookie example.com lifetime 3600

[AlexDaichendt]

@AlexDaichendt , also, try adding the following after cookie domain example.com.

cookie domain example.com
cookie example.com lifetime 3600

This did not change anything.

[Frando]

I think I have the same issue. I have two caddy servers, on sitea.org and siteb.org. sitea.org serves an auth portal. I want to reuse this auth portal in siteb.org. I have a shared key setup in crypto key sign-verify.

I have the following code in the Caddyfile for sitea.org:

security {
    authentication portal sitea {
            cookie domain sitea.org
            cookie domain siteb.org
            cookie sitea.org lifetime 900
            cookie siteb.org lifetime 900
   }
}

And then authorize with directives for sites on subdomains for both sitea.org and siteb.org. sitea.org works, but siteb.org goes into an endless redirect loop. When inspecting the requests and responses, I see that the cookie always has sitea.org and never siteb.org.

I was reading #43 which seems to enable this (as the official docs don't list the syntax above). Any ideas what's wrong?

Both caddy servers are using ghcr.io/authp/authp:v1.0.1 Docker image.

[bpas62]

Hi @greenpau , I use the latest versions of caddy and caddy-security and I encounter exactly the same problem.

My need:

• 1 authportal for 2 differents domains hosted on 2 differents caddy (1 public reverse-proxy and 1 private reverse-proxy)

My configuration is quite the same of @AlexDaichendt

Can you please tell me if this feature is supported?

I think I read the entire documentation without success.

Thanks for your help.

[greenpau]

@bpas62 , it would only work if you have a way to route requests to the same caddy instance for each user. The servers do not have shared state.

[bpas62]

Thank you for your quick response, it's clear to me. In my case, as the idea is to have 1 private reverse proxy and therefore isolated from the Internet, I do not want to route users through the public reverse proxy. I will therefore set up 2 authportals. Sincerely