Redirect Loop with caddy-security

[jshessen]

Describe the issue Appear to be in a redirect loop where the access_token is not being found

Configuration

Paste full Caddyfile below:

{
    ##############################
    # General Options
    #
    http_port 18080
    https_port 18443
    debug
    #
    ##############################

    ##############################
    # TLS Options
    #

    #Disable HTTPS/TLS
    #auto_https off
    local_certs
    #
    ##############################

    ##############################
    # caddy-security
    #

    # Set Order
    order authenticate before respond
    order authorize before basicauth

    security {
        # Local Identity Store
        local identity store localdb {
            realm local
            path /etc/caddy/users.json
        }
        authentication portal myportal {
            crypto default token lifetime 3600
            crypto key sign-verify {env.JWT_SHARED_KEY}
            #cookie domain .local
            enable identity store localdb
            ui {
                links {
                    "My Identity" "/whoami" icon "las la-user"
                    "Prometheus" "http://catacomb.local:9090" icon "las la-fire"
                    "Alert Manager" "http://catacomb.local:9093" icon "las la-exclamation-circle"
                    "Grafana" "http://catacomb.local:3000" icon "las la-tachometer-alt"
                }
                #password_recovery_enabled yes
            }
            transform user {
                match origin local
                action add role authp/admin
                action add role authp/user
                ui link "Portal Settings" /settings icon "las la-cog"
            }
        }

        authorization policy guests_policy {
            # disable auth redirect
            set auth url https://catacomb.local:18443/
            allow roles authp/admin authp/user
            crypto key verify {env.JWT_SHARED_KEY}
            acl rule {
                comment allow guests only
                match role guest authp/guest
                allow stop log info
            }
            acl rule {
                comment default deny
                match any
                deny log warn
            }
        }

        authorization policy users_policy {
            set auth url https://catacomb.local:18443/
            allow roles authp/admin authp/user
            crypto key verify {env.JWT_SHARED_KEY}
            acl rule {
                comment allow users
                match role authp/user
                allow stop log info
            }
            acl rule {
                comment default deny
                match any
                deny log warn
            }
        }

        authorization policy admins_policy {
            set auth url https://catacomb.local:18443/
            allow roles authp/admin authp/user
            crypto key verify {env.JWT_SHARED_KEY}
            acl rule {
                comment allow users
                match role authp/admin
                allow stop log info
            }
            acl rule {
                comment default deny
                match any
                deny log warn
            }
        }
    }
    #
    ##############################
}

##############################
# Snippets
#
(basic-auth) {
    basicauth /* {
        #pw = hiccup
        admin JDJhJDEwJEVCNmdaNEg2Ti5iejRMYkF3MFZhZ3VtV3E1SzBWZEZ5Q3VWc0tzOEJwZE9TaFlZdEVkZDhX
    }
}
#
##############################

##############################
# Sites
#

# Authentication Portal
catacomb.local:18443 {
    route {
        authenticate with myportal
    }
}

:{$PROMETHEUS_PORT} {
    route {
        authorize with admins_policy
        reverse_proxy prometheus:{$PROMETHEUS_PORT}
    }
}
:{$ALERTMANAGER_PORT} {
    route {
        import basic-auth
        reverse_proxy alertmanager:{$ALERTMANAGER_PORT}
    }
}
:{$GRAFANA_PORT} {
    route {
        trace tag="grafana" response_debug=yes
        reverse_proxy grafana:{$GRAFANA_PORT}
    }
}
#
##############################

Version Information

Provide output of caddy list-modules -versions | grep git below:

$ caddy_container_id=$(docker ps | grep caddy | awk '{print $1;}')
$ docker exec -w /etc/caddy $caddy_container_id caddy version
v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=

admin.api.load v2.5.1
admin.api.metrics v2.5.1
admin.api.pki v2.5.1
admin.api.reverse_proxy v2.5.1
caddy.adapters.caddyfile v2.5.1
caddy.config_loaders.http v2.5.1
caddy.listeners.http_redirect v2.5.1
caddy.listeners.tls v2.5.1
caddy.logging.encoders.console v2.5.1
caddy.logging.encoders.filter v2.5.1
caddy.logging.encoders.filter.cookie v2.5.1
caddy.logging.encoders.filter.delete v2.5.1
caddy.logging.encoders.filter.hash v2.5.1
caddy.logging.encoders.filter.ip_mask v2.5.1
caddy.logging.encoders.filter.query v2.5.1
caddy.logging.encoders.filter.regexp v2.5.1
caddy.logging.encoders.filter.rename v2.5.1
caddy.logging.encoders.filter.replace v2.5.1
caddy.logging.encoders.json v2.5.1
caddy.logging.writers.discard v2.5.1
caddy.logging.writers.file v2.5.1
caddy.logging.writers.net v2.5.1
caddy.logging.writers.stderr v2.5.1
caddy.logging.writers.stdout v2.5.1
caddy.storage.file_system v2.5.1
http v2.5.1
http.authentication.hashes.bcrypt v2.5.1
http.authentication.hashes.scrypt v2.5.1
http.authentication.providers.http_basic v2.5.1
http.encoders.gzip v2.5.1
http.encoders.zstd v2.5.1
http.handlers.acme_server v2.5.1
http.handlers.authentication v2.5.1
http.handlers.copy_response v2.5.1
http.handlers.copy_response_headers v2.5.1
http.handlers.encode v2.5.1
http.handlers.error v2.5.1
http.handlers.file_server v2.5.1
http.handlers.headers v2.5.1
http.handlers.map v2.5.1
http.handlers.metrics v2.5.1
http.handlers.push v2.5.1
http.handlers.request_body v2.5.1
http.handlers.reverse_proxy v2.5.1
http.handlers.rewrite v2.5.1
http.handlers.static_response v2.5.1
http.handlers.subroute v2.5.1
http.handlers.templates v2.5.1
http.handlers.tracing v2.5.1
http.handlers.vars v2.5.1
http.matchers.expression v2.5.1
http.matchers.file v2.5.1
http.matchers.header v2.5.1
http.matchers.header_regexp v2.5.1
http.matchers.host v2.5.1
http.matchers.method v2.5.1
http.matchers.not v2.5.1
http.matchers.path v2.5.1
http.matchers.path_regexp v2.5.1
http.matchers.protocol v2.5.1
http.matchers.query v2.5.1
http.matchers.remote_ip v2.5.1
http.matchers.vars v2.5.1
http.matchers.vars_regexp v2.5.1
http.precompressed.br v2.5.1
http.precompressed.gzip v2.5.1
http.precompressed.zstd v2.5.1
http.reverse_proxy.selection_policies.cookie v2.5.1
http.reverse_proxy.selection_policies.first v2.5.1
http.reverse_proxy.selection_policies.header v2.5.1
http.reverse_proxy.selection_policies.ip_hash v2.5.1
http.reverse_proxy.selection_policies.least_conn v2.5.1
http.reverse_proxy.selection_policies.random v2.5.1
http.reverse_proxy.selection_policies.random_choose v2.5.1
http.reverse_proxy.selection_policies.round_robin v2.5.1
http.reverse_proxy.selection_policies.uri_hash v2.5.1
http.reverse_proxy.transport.fastcgi v2.5.1
http.reverse_proxy.transport.http v2.5.1
http.reverse_proxy.upstreams.a v2.5.1
http.reverse_proxy.upstreams.srv v2.5.1
pki v2.5.1
tls v2.5.1
tls.certificates.automate v2.5.1
tls.certificates.load_files v2.5.1
tls.certificates.load_folders v2.5.1
tls.certificates.load_pem v2.5.1
tls.certificates.load_storage v2.5.1
tls.get_certificate.http v2.5.1
tls.get_certificate.tailscale v2.5.1
tls.handshake_match.remote_ip v2.5.1
tls.handshake_match.sni v2.5.1
tls.issuance.acme v2.5.1
tls.issuance.internal v2.5.1
tls.issuance.zerossl v2.5.1
tls.stek.distributed v2.5.1
tls.stek.standard v2.5.1

  Standard modules: 96

dns.providers.cloudflare v0.0.0-20210607183747-91cf700356a1
http.authentication.providers.authorizer v1.1.14
http.handlers.authenticator v1.1.14
http.handlers.trace v1.1.10
security v1.1.14

  Non-standard modules: 5

  Unknown modules: 0

Expected behavior

Describe expected behavior. Expect that after initial token failure, the authentication portal will prompt, and then redirect to original requested URL

Additional context

Add any other context about the problem here.

[greenpau]

@jshessen ,

# Authentication Portal
catacomb.local:18443 {
    route {
        authenticate with myportal
    }
}

Please change the above to:

# Authentication Portal
catacomb.local:18443 {
   authenticate with myportal
}

Please delete the following. You are not using these policies. It will make the config smaller.

        authorization policy guests_policy {
            # disable auth redirect
            set auth url https://catacomb.local:18443/
            allow roles authp/admin authp/user
            crypto key verify {env.JWT_SHARED_KEY}
            acl rule {
                comment allow guests only
                match role guest authp/guest
                allow stop log info
            }
            acl rule {
                comment default deny
                match any
                deny log warn
            }
        }

        authorization policy users_policy {
            set auth url https://catacomb.local:18443/
            allow roles authp/admin authp/user
            crypto key verify {env.JWT_SHARED_KEY}
            acl rule {
                comment allow users
                match role authp/user
                allow stop log info
            }
            acl rule {
                comment default deny
                match any
                deny log warn
            }
        }

When you browse to catacomb.local:18443, do you get a login screen?

[jshessen]

Updated to remove the "guest" and "user" policies and removed the route in the authentication site as requested.

Yes, I get a redirect to the login, and can invoke both good (redirect loop) and bad (sandbox) login credentials.

I can also interact with the portal after login at this URL https://catacomb.local:18443/portal

[greenpau]

@jshessen , the token you get from auth portal is granted to you via HTTPS.

The links below suggest you are using using HTTP. If that the case the token issued via HTTPS will not be passed to HTTP.

                    "Prometheus" "http://catacomb.local:9090" icon "las la-fire"
                    "Alert Manager" "http://catacomb.local:9093" icon "las la-exclamation-circle"
                    "Grafana" "http://catacomb.local:3000" icon "las la-tachometer-alt"

Option 1: Add the following cookie directives:

authentication portal myportal {
  cookie insecure on
  cookie strip domain
  ...
}

Option 2: (better) Proxy Prometheus, Alert Manager, and Grafana via 18443.

# Authentication Portal
catacomb.local:18443 {
    route /auth* {
        authenticate with myportal
    }
        route /prometheus* {
                authorize with admins_policy
        reverse_proxy prometheus:{$PROMETHEUS_PORT}
        } 
        route /grafana* {
                authorize with admins_policy
        reverse_proxy prometheus:{$GRAFANA_PORT}
        }
    route /alertmanager* {
                authorize with admins_policy
        reverse_proxy alertmanager:{$ALERTMANAGER_PORT}
    }
}

[greenpau]

If that the case the token issued via HTTPS will not be passed to HTTP.

@jshessen , more on it here:

• https://owasp.org/www-community/controls/SecureCookieAttribute
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies

[jshessen]

@greenpau this was a perfect answer! (been banging around for 2 days now)

I implemented Option 1 to get a quick response. I am looking to expand this across my environment (very subdomain heavy) however, I can look to modify the fully caddy sites (reverse proxies) to be HTTPS/TLS to maintain the cookie integrity to the non-TLS backends as I move forward

[jshessen]

I had tried setting up TLS on the subsites, but had attempted to reduce the configuration as an act of troubleshooting, I should be able to go back to expanding the config.

Thank you for the prompt response.

[greenpau]

@jshessen , more specifically on Prom and Alert Manager:

  route /prometheus* {
    authorize with admins_policy
    uri strip_prefix /prometheus
    reverse_proxy http://127.0.0.1:9080
  }

  route /alertmanager* {
    authorize with admins_policy
    uri strip_prefix /alertmanager
    reverse_proxy http://127.0.0.1:9083
  }

The config file for Prometheus should containweb.external-url:

cat << EOF > ${MYAPP_CONF}
OPTIONS="--log.level=debug \
--config.file=/etc/prometheus/config/prometheus.yml \
--storage.tsdb.path=/opt/data/prometheus \
--web.console.templates=/var/lib/prometheus/consoles \
--web.console.libraries=/var/lib/prometheus/console_libraries \
--web.external-url https://${HOSTNAME}/prometheus"
EOF

Same for AlertManager:

cat << EOF > ${MYAPP_CONF}
OPTIONS="--log.level=debug \
--config.file=/etc/alertmanager/config/alertmanager.yml \
--storage.path=/opt/data/alertmanager \
--data.retention=1440h \
--web.external-url https://${HOSTNAME}/alertmanager \
--web.route-prefix=/"
EOF

[greenpau]

@jshessen , if you want, please connect with me on LinkedIn and I will help you with your setup. We could do Google Meet or Zoom.

[jshessen]

@greenpau, thank you again ... I will try to find a window here in the next few weeks to reach out.

[greenpau]

this was a perfect answer! (been banging around for 2 days now)

@jshessen , next time open an issue after 15 minutes!

[jshessen]

@greenpau gave me a chance to really dig through all the different combinations of templates and documents!

[greenpau]

@jshessen , actually, prom/alertmanager/grafana was my first use case for this plugin 😄

[greenpau]

@jshessen , you might also be interested in https://github.com/greenpau/caddy-security/issues/105#issuecomment-1129961604

Basically, when you proxy traffic to Grafana, you could set auth level via X-Auth-Level.

Also, see X-WEBAUTH-USER docs: https://grafana.com/docs/grafana/next/setup-grafana/configure-security/configure-authentication/auth-proxy/#interacting-with-grafanas-authproxy-via-curl

[greenpau]

@jshessen , I am looking to add testimonial sections to https://authcrunch.com. Could you please write one and send it to me at greenpau@outlook.com?