Cross-tenant requests are possible

[markbanierink]

We have a multi-tenant setup where based on the subdomain in the URL a tenant is determined. Each tenant has his own Keycloak realm where he can authenticate. So when you do a request to tenanta.ourdomain.com, you authenticate at realm "tenanta" in Keycloak. We use the default setting with cookies as session storage. When you authenticate at tenanta, you get a cookie that holds the session information. However, if I do a request to tenantb and add this cookie of tenanta to the request, I am still authenticated.

Environment

• lua-resty-openidc 1.7.6
• nginx 1.27.0
• Keycloak 25.0.4

Configuration

nginx.conf:

server {
    # Example: tenanta.ourdomain.com
    server_name ~ ^ (?!www\.)^(?<tenant>.+)\.ourdomain.com$;

    set $session_secret very_secret;

    # some more configuration..

    access_by_lua_block {
        require("authentication").authenticate()
    }

    # some more configuration..
}

authentication.lua:

function authentication.authenticate()
    local base_url = ngx.var.scheme .. "://" .. ngx.var.http_host
    local tenant = ngx.var.tenant

    local opts = {
        discovery = base_url .. "/auth/realms/" .. tenant .. "/.well-known/openid-configuration"
        -- some more options..
    }

    local res, err = openidc.authenticate(opts)

    -- some more code..
end

Minimized example

1. Create two realms in Keycloak named "tenanta" and "tenantb".
2. Do a request to tenantA at tenanta.ourdomain.com and authenticate.
3. Do the same request, but change the url, host header and origin header to tenantb.ourdomain.com (used Burp Suite).

Expected behaviour

Step 3 resulting in a failed request, responding with a HTTP 302 or 401, depending on the configuration.

Actual behaviour

Step 3 responds with the requested resource, authentication went fine.

Additional

We figured we might make the session secret tenant specific, like in this nginx configuration:

set $session_secret very_secret${tenant};

Any request to tenantA results in a cookie encrypted with a session secret specific for tenantA, since it is part of the secret. A request like the one in step 3 would, when requesting to tenantB, then be decrypted with the wrong secret and automatically the request would fail. This seemed to work at moments, but a lot of times it just authenticated fine for tenantB. So far we couldn't figure out what caused this unexpected behaviour.

Possible solutions:

1. Add a check that compares the issuer of the cookie data with the current opts.
2. Fix the tenant specific session secret mechanism.
3. ?

[bodewig]

Does using opts.cache_segment help in this situation? See https://github.com/zmartzone/lua-resty-openidc?tab=readme-ov-file#caching-of-introspection-and-jwt-verification-results and #399

[markbanierink]

No, it doesn't help. I've added it to my options like this cache_segment = tenant, where tenant is the variable for the tenant name and I can still do cross-tenant requests. This is only possible when using the openidc.authenticate() function. In another flow where we use openidc.bearer_jwt_verify() this is not possible and everything works correctly.

[oldium]

I think you need to limit the cookie scope to subdomain and also verify that the cookie is used on the correct subdomain afterwards somehow, because malicious user could take cookie from tenanta.ourdomain.com and use it on tenantb.ourdomain.com. I think the code simply has no knowledge that the cookie from tenanta is wrong when used on tenantb.

[oldium]

Just an untested idea:

According to https://github.com/bungle/lua-resty-session (lua-resty-session 4 is required, so master branch of lua-resty-openidc is needed) it should be possible to set cookie audience to your tenant for example. Then your code will be responsible to open the session with the correct audience (require "resty.session".open({audience = tenant,})) and you will pass it to lua-resty-openidc as a session_or_opts argument.

The cookie domain can be controlled with cookie_domain parameter, so at the end you could use someting like require "resty.session".open({audience = tenant, cookie_domain=tenant_fqdn})) to open the correct session. lua-resty-openidc will save any session changes.

For the bearer_jwt_verify flow you also need to verify that the JWT is used on the correct domain, but that could be part of other access rights check (same JWT could be used cross-site).

EDIT: openidc.authenticate(opts, target_url, unauth_action, {audience = tenant, cookie_domain=tenant_fqdn}) should work too and lua-resty-openidc will open the correct session.

cookie_domain looks undocumented, but it is in the code of lua-resty-session 😊

[markbanierink]

@zandbelt @oldium @bodewig I have just tried openidc.authenticate(opts, target_url, unauth_action, {audience = tenant, cookie_domain=tenant_fqdn}) with the master lua-resty-openidc with session 4.0.5 and it seems to prevent cross-tenant requests from being possible 👍. Nice, and thank you all for the help!

[oldium]

Just to be sure to prevent any surprises - the tenant_fqdn variable used in my code snippet is not present in your original code, so it needs to be set to the fully qualified domain name of the tenant (the tenant's top-level domain), i.e. tenanta.ourdomain.com.

[oldium]

One additional note - omitting the cookie_domain entirely is not an error, it depends on what you want:

1. Without cookie_domain (or with cookie_domain=nil), the cookie will be set only for the request's domain (which is hopefully tenanta.ourdomain.com anyway) and not its subdomains.
2. With cookie_domain="tenanta.ourdomain.com", the cookie will be additionally available also for subdomains, i.e. service.tenanta.ourdomain.com.
3. With cookie_domain="ourdomain.com", you still get a working cookie separation, but the cookie value will be shared with all tenants. This works because the cookie data has a key-value map and the audience acts like a key to get the audience-specific data (i.e. tenant-specific data in your case). In order to have this working, the session's ikm/secret used to derive cookie data AES encryption key has to be the same for all tenants, otherwise some tenants will be unable to decode the cookie value and get the audience-specific data. Please check lua-resty-session documentation for more info about data encryption. The cookie value will also be bigger in this case, because it will contain all tenants' data, so I do not recommend this.

[systemofapwne]

Not sure, if this is related, but since the recent update to 1.8 I am getting CORS related errors once my keycloak session needs to be refreshed.

Here for example, grafana wants to pull fresh data but the OIDC session needs to be refreshed. So resty-openid redirects the browser to the keycloak auth backend and then the browser tries to preflight that domain for CORS rules via the 'OPTIONS' method. This then fails with 405 (Not allowed)

Besides of the recent upgrade to resty-openidc 1.8, I did not change my setup:

[zandbelt]

this then looks like something caused by lua-resty-session's behavioral changes between <=3.x and >= 4.x; it may be a setting that is configurable, hopefully someone else can answer that here or you could ask the question in the lua-resty-session project

[systemofapwne]

this then looks like something caused by lua-resty-session's behavioral changes between <=3.x and >= 4.x; it may be a setting that is configurable, hopefully someone else can answer that here or you could ask the question in the lua-resty-session project

This makes sense. I will try to further investigate this in the near future. For the time being, I downgraded to 1.7.6-3 and all my problems went away.

[oldium]

this then looks like something caused by lua-resty-session's behavioral changes between <=3.x and >= 4.x; it may be a setting that is configurable, hopefully someone else can answer that here or you could ask the question in the lua-resty-session project

That is absolutely correct. There are new defaults for the lua-resty-session library, especially the following options:

cookie_path="/"
cookie_http_only=true
cookie_same_site="Lax"

This is a behaviour change, so the Set-Cookie header with lua-resty-session 4.x adds the following values when the default session configuration is used:

...; Path=/; SameSite=Lax; HttpOnly; ...

And a note - one interesting option is not yet documented, which is cookie_domain.

[systemofapwne]

this then looks like something caused by lua-resty-session's behavioral changes between <=3.x and >= 4.x; it may be a setting that is configurable, hopefully someone else can answer that here or you could ask the question in the lua-resty-session project

That is absolutely correct. There are new defaults for the lua-resty-session library, especially the following options:

cookie_path="/"
cookie_http_only=true
cookie_same_site="Lax"

This is a behaviour change, so the Set-Cookie header with lua-resty-session 4.x adds the following values when the default session configuration is used:

...; Path=/; SameSite=Lax; HttpOnly; ...

And a note - one interesting option is not yet documented, which is cookie_domain.

This looks like I only need to set the Same-Site policy from Lax to None to make subrequest working reliable again, I guess. I just wonder about the security impact here. Setting session_domain cookie_domain to "domain.tld" (which then became ".domain.tld" for the cookie) didn't work.

[oldium]

I gave this second look with a great help of internet (MDN/CORS, RFC 6749...). Is this really a cookie/session problem? The question is what really happened...

The URL we see in the screenshot is the auth endpoint, which means that it is the start of the Authorization Code Grant flow. This means that the token refresh failed for some reason, so you were redirected to the auth endpoint.

We see the OPTIONS with Access-Control-Request-* headers to the auth endpoint, meaning this is Preflight request. Preflight requests are meant to protect malicious scripts from accessing different sites, so this had to be actually a JavaScript XHR request. But the auth endpoint is meant to be loaded only by the browser as a web page intentionally, not as an API (via XHR), so this is why the CORS request failed.

So Grafana tried to refresh shown data with background XHR request, the attempt to refresh the access token was made (or not?) and failed, so the full auth redirect was sent, which failed because it should be a browser page redirect, not XHR redirect.

So it would be interesting to check why the token refresh failed actually.

And one note: you probably have typo in your last comment, it should be cookie_domain, not session_domain.

[systemofapwne]

@oldium Your analysis of the situation seems about on point. I myself do not understand, why this happens and unfortuantely, my workaround with SameSite = None did not work for long. Something else is off.

[oldium]

It would be good if you would be able to collect DEBUG logs from Nginx running lua-resty-openidc.

[oldium]

I just tested it with the mock OIDC server (I faked expires_in to be 10 seconds in the token endpoint to test refresh) and the token refresh request and response looks good to me. This is how it looks like in Kong GW logs with our custom Kong GW auth plugin using lua-resty-openidc, so that you can compare by yourself.

[lua] openidc.lua:1426: openidc_access_token(): refreshing expired access_token: eyJraWQiOiJhMTZlYTUzNGFjYWY3MTdmY2I1ODVlYjZiNjVkMjFlOGQxMjE5ZDFjMjZkZjNhYmRjYzc5ODMxNzliMDVhYjNjMWQ3MGUxMzYyNTgxOTE2NyIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkwODAiLCJpYXQiOjE3MjczMzQ5MTQsImV4cCI6MTcyNzMzODUxNCwibmJmIjoxNzI3MzM0OTA0LCJzdWIiOiJhZG1pbl91c2VyIiwiYW1yIjpbInB3ZCJdLCJzY29wZSI6Im9wZW5pZCByb2xlcyJ9.U8pCASAFSUGnJ6XURTZTJRz8scYz1jvFHShDoNNd_72Gbd-WagGGrvcF-FS9N_EfIeRObl3ge3xatPqvcSGhlBGjUHxcgKiBD7q8bR0Tbeg07o6AJs3GoZVDh6F0KuB0VZfma4Vpm8aWIJqsruK1xi0qOiydC_kjkiKztPLxTdqPZuYHMmD40D8NTtnqxFdm8xQl2qXX_GhLn3UeJMQ68-bZER9OZ5sjarX5hi1kUZBRaAywvYFzHUeq5JRe_y1S8ZAMeBV1jjsFHIEPXqrZW93oFiOfAilUhe5OxS889t7z2m7u3jQVRLUkgFkpIGtIvhWxSrRbThYR_Eq8MInZMQ with: 071db3bd-f479-4344-b00d-52db4fb8d4a4
[lua] openidc.lua:701: openidc_get_token_auth_method(): token_endpoint_auth_method result set to client_secret_basic
[lua] openidc.lua:455: call_token_endpoint(): client_secret_basic: authorization header 'Basic dGVzdF9jbGllbnQ6dGVzdF9zZWNyZXQ='
[lua] openidc.lua:510: call_token_endpoint(): request body for token endpoint call: scope=openid%20roles&grant_type=refresh_token&refresh_token=071db3bd-f479-4344-b00d-52db4fb8d4a4
[lua] openidc.lua:429: openidc_configure_proxy(): openidc_configure_proxy : don't use http proxy
[lua] http_connect.lua:253: execute_original_func(): poolname: http:openid-test-server.home.arpa:9080:nil::nil:::
[lua] openidc.lua:528: call_token_endpoint(): token endpoint response: {"access_token":"eyJraWQiOiJhMTZlYTUzNGFjYWY3MTdmY2I1ODVlYjZiNjVkMjFlOGQxMjE5ZDFjMjZkZjNhYmRjYzc5ODMxNzliMDVhYjNjMWQ3MGUxMzYyNTgxOTE2NyIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkwODAiLCJpYXQiOjE3MjczMzQ5NDAsImV4cCI6MTcyNzMzODU0MCwibmJmIjoxNzI3MzM0OTMwLCJzdWIiOiJhZG1pbl91c2VyIiwiYW1yIjpbInB3ZCJdLCJzY29wZSI6Im9wZW5pZCByb2xlcyJ9.TvsY3nAM4jNIjzG_6HVDTGCRKtnlN_2nyAi04zKZP7Z0BLgXxlAEVnHmOTA0nIn7eWfzO-_eUWU0B209mRiDkuU3L1d9_eG2pxqbzO35h37422z1gfDXqA_c2ZBnRKlszJb1b60jgxRjGbg2ToXcHIFp0u74vDgP0F8mM9D33Ff7iatQ_rzhZDtrEqcATL40XNWiOycgp26R7TcEk8ethMXX_7yotA8MAfJLVUTMJxVQzQ3Def49T74-BQ_jfyJHyA57JkgEWXU8BBVZ1wdEY2bOisHLSrGBFDUMItFINy2TO_6bUdN-nkcdhPJ_W5BEU01RRucqz0owRrYrhm-Ipw","token_type":"Bearer","expires_in":10,"scope":"openid roles","id_token":"eyJraWQiOiJhMTZlYTUzNGFjYWY3MTdmY2I1ODVlYjZiNjVkMjFlOGQxMjE5ZDFjMjZkZjNhYmRjYzc5ODMxNzliMDVhYjNjMWQ3MGUxMzYyNTgxOTE2NyIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkwODAiLCJpYXQiOjE3MjczMzQ5NDAsImV4cCI6MTcyNzMzODU0MCwibmJmIjoxNzI3MzM0OTMwLCJzdWIiOiJhZG1pbl91c2VyIiwiYXVkIjoidGVzdF9jbGllbnQiLCJzY29wZSI6Im9wZW5pZCByb2xlcyJ9.h7xx22vosopQWwayJm_6j1i02QI1I3wO8mDBHcOoRah5kMpU2ydAF8ClDvixRXYf8vfKt7cSH9M5NKWtgGQ0vjEhyNGU59ApAPt2b7nr2AQ9dvGgrJJELpB-28oKhT21OarjHuizRbI7Vdy2XoFm08s-KbzUSJ8jiaroKqhVVI392TONcBswdCpjaKcFb64B7xQ0XJqqyNy6J5B3PV5AS-eI2HHnzn11uAvO56ISP-GFzhCWWKw6244nBt4k5AWduC7KTI3DkGtAEDWvTMf9M2Z8fWtSiOSNBtQvS4zxoCLdoPjzB2VjM-WfIyXZVez7PnXrxaylh8qwYMl_R4WRUw","refresh_token":"b999870f-8e00-465c-ba94-c60a803c01ae"}
[lua] openidc.lua:1012: openidc_load_jwt_and_verify_crypto(): using discovery to find key
[lua] openidc.lua:117: openidc_cache_get(): cache hit: type=jwks key=http://openid-test-server.home.arpa:9080/jwks#a16ea534acaf717fcb585eb6b65d21e8d1219d1c26df3abdcc7983179b05ab3c1d70e13625819167
[lua] openidc.lua:1040: openidc_load_jwt_and_verify_crypto(): jwt: {"valid":true,"reason":"everything is awesome~ :p","raw_header":"eyJraWQiOiJhMTZlYTUzNGFjYWY3MTdmY2I1ODVlYjZiNjVkMjFlOGQxMjE5ZDFjMjZkZjNhYmRjYzc5ODMxNzliMDVhYjNjMWQ3MGUxMzYyNTgxOTE2NyIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0","signature":"h7xx22vosopQWwayJm_6j1i02QI1I3wO8mDBHcOoRah5kMpU2ydAF8ClDvixRXYf8vfKt7cSH9M5NKWtgGQ0vjEhyNGU59ApAPt2b7nr2AQ9dvGgrJJELpB-28oKhT21OarjHuizRbI7Vdy2XoFm08s-KbzUSJ8jiaroKqhVVI392TONcBswdCpjaKcFb64B7xQ0XJqqyNy6J5B3PV5AS-eI2HHnzn11uAvO56ISP-GFzhCWWKw6244nBt4k5AWduC7KTI3DkGtAEDWvTMf9M2Z8fWtSiOSNBtQvS4zxoCLdoPjzB2VjM-WfIyXZVez7PnXrxaylh8qwYMl_R4WRUw","header":{"kid":"a16ea534acaf717fcb585eb6b65d21e8d1219d1c26df3abdcc7983179b05ab3c1d70e13625819167","alg":"RS256","typ":"JWT"},"payload":{"exp":1727338540,"scope":"openid roles","sub":"admin_user","iss":"http://localhost:9080","aud":"test_client","nbf":1727334930,"iat":1727334940},"verified":true,"raw_payload":"eyJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkwODAiLCJpYXQiOjE3MjczMzQ5NDAsImV4cCI6MTcyNzMzODU0MCwibmJmIjoxNzI3MzM0OTMwLCJzdWIiOiJhZG1pbl91c2VyIiwiYXVkIjoidGVzdF9jbGllbnQiLCJzY29wZSI6Im9wZW5pZCByb2xlcyJ9"} ,valid: true, verified: true
[lua] openidc.lua:1083: openidc_load_and_validate_jwt_id_token(): id_token header: {"kid":"a16ea534acaf717fcb585eb6b65d21e8d1219d1c26df3abdcc7983179b05ab3c1d70e13625819167","alg":"RS256","typ":"JWT"}
[lua] openidc.lua:1084: openidc_load_and_validate_jwt_id_token(): id_token payload: {"exp":1727338540,"scope":"openid roles","sub":"admin_user","iss":"http://localhost:9080","aud":"test_client","nbf":1727334930,"iat":1727334940}
[lua] openidc.lua:1454: openidc_access_token(): access_token refreshed: eyJraWQiOiJhMTZlYTUzNGFjYWY3MTdmY2I1ODVlYjZiNjVkMjFlOGQxMjE5ZDFjMjZkZjNhYmRjYzc5ODMxNzliMDVhYjNjMWQ3MGUxMzYyNTgxOTE2NyIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjkwODAiLCJpYXQiOjE3MjczMzQ5NDAsImV4cCI6MTcyNzMzODU0MCwibmJmIjoxNzI3MzM0OTMwLCJzdWIiOiJhZG1pbl91c2VyIiwiYW1yIjpbInB3ZCJdLCJzY29wZSI6Im9wZW5pZCByb2xlcyJ9.TvsY3nAM4jNIjzG_6HVDTGCRKtnlN_2nyAi04zKZP7Z0BLgXxlAEVnHmOTA0nIn7eWfzO-_eUWU0B209mRiDkuU3L1d9_eG2pxqbzO35h37422z1gfDXqA_c2ZBnRKlszJb1b60jgxRjGbg2ToXcHIFp0u74vDgP0F8mM9D33Ff7iatQ_rzhZDtrEqcATL40XNWiOycgp26R7TcEk8ethMXX_7yotA8MAfJLVUTMJxVQzQ3Def49T74-BQ_jfyJHyA57JkgEWXU8BBVZ1wdEY2bOisHLSrGBFDUMItFINy2TO_6bUdN-nkcdhPJ_W5BEU01RRucqz0owRrYrhm-Ipw updated refresh_token: b999870f-8e00-465c-ba94-c60a803c01ae
[lua] openidc.lua:1578: authenticate(): session_present=true, session.data.id_token=false, session.data.authenticated=true, opts.force_reauthorize=nil, opts.renew_access_token_on_expiry=nil, try_to_renew=true, token_expired=false

Please be aware that the debug logs contain sensitive information (authorization header with password, access token, refresh token), so do not share it plain text without modifications to anybody!