Session cookies never expires

[abompard]

According to the GssapiUseSessions documentation, the session cookies should expire according to the lifetime of the GSSAPI session established at authentication. I don't see the expiration beeing set in the cookie header:

$ curl -v -u : --negotiate https://fasjson.fedoraproject.org/v1/me/
[...]
< HTTP/2 200 
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< www-authenticate: Negotiate [...]
< set-cookie: ipa_session=MagBearerToken=UXmr[...]Gwo%3d;path=/;httponly;secure;
< set-cookie: 258ec7ac3fe42ca4f3a9165f864d24b3=50374418bc7687d83d82fe30a6c36ce4; path=/; HttpOnly; Secure; SameSite=None
< apptime: D=319790
< 
{"result": {"dn": "uid=abompard,cn=users,cn=accounts,dc=fedoraproject,dc=org", "username": "abompard", "service": null, "uri": "https://fasjson.fedoraproject.org/v1/users/abompard/"}}

My config file includes:

GssapiUseSessions On
Session On
SessionCookieName ipa_session path=/;httponly;secure;
SessionHeader IPASESSION
GssapiSessionKey file:/httpdir/run/session.key

If I look at the ipa_session cookie header sent back to curl, I don't see any Expires attribute. I think that may be why my long-running http client end up getting 401's: they keep the session cookie around when they should drop it.

[simo5]

On a 401 the client should re-attempt authentication and get a new cookie, not sure why that would not be working. Adding an expiration on the cookie can probably be done, but would rather complicate matters and risk other sync issue.

[abompard]

I am currently testing another hypothesis, I had set the GssapiSessionKey to a file that does not exist on startup, so according to the docs mod_auth_gssapi will create a new one on startup. But it's in a temp storage, so if the pod is restarted, the file will change and clients with long-running sessions will suddenly see their session invalidated. I think that's why I'm getting 401s with Credential lifetime has expired (I've tested the credentials, they aren't expired). The client is in Python, based on requests-gssapi.

[abompard]

I was wrong again, the cause of the 401 is that the credentials made available in GssapiDelegCcacheDir on the server have expired:

$ klist FILE:/httpdir/run/ccaches/bugzilla2fedmsg~os-control01.iad2.fedoraproject.org\@FEDORAPROJECT.ORG 
Ticket cache: FILE:/httpdir/run/ccaches/bugzilla2fedmsg~os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG
Default principal: bugzilla2fedmsg/os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG

Valid starting     Expires            Service principal
09/16/24 23:58:01  09/17/24 23:58:01  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
        for client HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG
09/17/24 08:49:46  09/18/24 08:47:13  HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG
        renew until 09/24/24 08:47:13
09/17/24 08:49:46  09/17/24 23:58:01  ldap/ipa02.iad2.fedoraproject.org@
        Ticket server: ldap/ipa02.iad2.fedoraproject.org@FEDORAPROJECT.ORG
09/17/24 08:49:46  09/17/24 23:58:01  ldap/ipa03.iad2.fedoraproject.org@
        Ticket server: ldap/ipa03.iad2.fedoraproject.org@FEDORAPROJECT.ORG
09/17/24 08:49:46  09/17/24 23:58:01  ldap/ipa01.iad2.fedoraproject.org@
        Ticket server: ldap/ipa01.iad2.fedoraproject.org@FEDORAPROJECT.ORG

Is there something my application should do to renew them? The credentials on the client still had a lifetime of 59536s when the delegated credential on the server was found expired and the 401 was returned. Could it be that using sessions prevents the client from renewing the delegated credential on the server? Or are sessions unrelated? I'm not sure what's going on here, and what I'm missing.

[simo5]

Ideally you have a session key has a shorter expiration than credentials, when using sessions the client will just try to send a session cookie, however upon receiving a 401 the client should simply try a new gssapi authentication, did the client simply send back the session cookie and not try to perform an actual authentication ? Or was auth attmpted and something went wrong ?

The web server log may have some good pointers if you enable the debug level.

[abompard]

The server code checks the ticket lifetime of the credentials that mod_auth_gssapi makes available in KRB5CCACHE (in /httpdir/run/ccaches), find that they are expired by looking at their lifetime attribute. If it's expired, it'll return a 401 response, but it's a regular 401 response without any "Negotiate" header, maybe the client expects it to retry authentication (at least that's what I understand from requests-gssapi's code)?

Is there a way for my server python code to trigger a proper 401 response from mod_auth_gssapi with the Negotiate header when the delegated credentials are expired?

[simo5]

I think 401 with Negotiate header is the regular way to ask a client to authenticate, why is the client not trying is the question ...

[abompard]

Hmm no I'm not sending a Negotiate header to the client when I do a 401. I "manually" do a 401 from the Python code when I see that the delegated credentials have expired. And I can't send the Negotiate header because this is all handled at the mod_auth_gssapi level. I it normal that the delegated credentials that mod_auth_gssapi provides my application with can be expired? Shouldn't it see that they are expired and ask the client to re-authenticate? Or am I missing something else?

[simo5]

If mod_auth_gssapi sees that the creds expiration has been reached in mag_check_session() then it will not proceed to the application at all and will return a 401 negotiate.

However if you return 401 directly from the application then mod_auth_gssapi is not involved anymore and you will have to return the Negotiate header yourself, alternatively you could return a Redirect error to the client where you also tell the client to invalidate the cookie. This will caus an additional rountrip in the client but will allow mod_auth_gssapi to send the 401 itself with the correct negotiate headers.

What is odd is that it seem there is a disagreement between mod_auth_gssapi and your application on the expiration time of the creds??

[abompard]

alternatively you could return a Redirect error to the client where you also tell the client to invalidate the cookie.

Oh good idea, I'll try that.

What is odd is that it seem there is a disagreement between mod_auth_gssapi and your application on the expiration time of the creds??

Indeed! Here's the delegated credentials that mod_auth_gssapi provides my application with:

$ klist -c /httpdir/run/ccaches/bugzilla2fedmsg~os-control01.iad2.fedoraproject.org\@FEDORAPROJECT.ORG
Ticket cache: FILE:/httpdir/run/ccaches/bugzilla2fedmsg~os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG
Default principal: bugzilla2fedmsg/os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG

Valid starting     Expires            Service principal
09/23/24 07:56:02  09/24/24 07:56:02  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
        for client HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG
09/23/24 09:06:17  09/24/24 09:06:16  HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG
        renew until 09/30/24 09:06:16
09/23/24 09:06:17  09/24/24 07:56:02  ldap/ipa02.iad2.fedoraproject.org@
        Ticket server: ldap/ipa02.iad2.fedoraproject.org@FEDORAPROJECT.ORG
09/23/24 09:06:17  09/24/24 07:56:02  ldap/ipa03.iad2.fedoraproject.org@
        Ticket server: ldap/ipa03.iad2.fedoraproject.org@FEDORAPROJECT.ORG
09/23/24 09:06:18  09/24/24 07:56:02  ldap/ipa01.iad2.fedoraproject.org@
        Ticket server: ldap/ipa01.iad2.fedoraproject.org@FEDORAPROJECT.ORG

I had to restart the client this morning because the authentication was expired again, but if it unfolds as before this time around again, I can tell that at 09/24/24 07:56:02, the session will still be valid, mod_auth_gssapi will not return 401, but my application will end up with an expired delegated credential and will not be able to talk to IPA over LDAP. Unfortunately I don't know C so I can't help with the code, but I'm happy to provide all the necessary info or run tests.

[abompard]

OK I did manage to workaround the issue by having the server return a 302 to the same address with the HTTP header that I had set in SessionHeader set to "MagBearerToken=". This invalidates the session, and the client just has to follow the redirect. Unfortunately, our client is based on a library that interprets the Swagger/OpenAPI spec very strictly, and does not follow redirects. There's (what I think is) a bug in there too that makes it not trivial to tell it to follow redirects. Changing that client-side code is possible but not easy to deploy as it would mean updating all the apps that use it, which all have their own lifecycles. At the moment, if the server sends them a 302, they'll just traceback, which is not ideal. So I'm interested in trying to fix this at the mod_auth_gssapi level, if we agree there's a bug in there about the session lifetime.

[simo5]

Perhaps I need to cross check the cache liftime with the lifetime claimed in the cookie. I expect there may be cases when the client can get confused and send a cookie for a lifetime that differs from the latest cached credentials ... I am not sure how that could happen, and I do not see any special security considerations to it. I am just not sure I can easily check for creds lifetime in the session handling code though. And I have no idea how I would test it ...

[simo5]

Ah but I thik I know how something like that can happen now that I think about your situation. Are you, by chance, sharing the same krb principal among multiple different clients?

That could cause a client that is configured to obtain a shorter lived credentials to overwrite the cache creds with ones that are shorter lived than the creds another client originally used to obtain their cookie.

Like:

Client A fetches krb ticket with 24h expiration time
Client A auths to your server and obtain session cookie with exp timestamp 24h in the future
Server stores cached creds valid 24h
... one hour later...
Client B fetches krb tikcet with 10h expiration time
Client A auths to your server and obtain session cookie with exp timestamp 10h in the future
Server stores cached creds valid 10h (overwriting the previous ones valid 24h)
... 10 hours later ...
ClientA contacts server with its session cookie that says there are still 13h of validity to spare .. but the creds are actually expired due to substitution.

Can you confirm if this is a scenario that may be happening in your environment?

[abompard]

Perhaps I need to cross check the cache liftime with the lifetime claimed in the cookie. I expect there may be cases when the client can get confused and send a cookie for a lifetime that differs from the latest cached credentials ...

When I looked at the cookie that the client receives, I didn't see any expiration date set. This is the received header:

'set-cookie': 'ipa_session=MagBearerToken=b1qD[...snip...]6sa8fw%3d%3d;path=/;httponly;secure;'

Ah but I thik I know how something like that can happen now that I think about your situation. Are you, by chance, sharing the same krb principal among multiple different clients?

I don't think so, the credential comes from a keytab that only this app is using, and there's only one concurrent pod running.

I've managed to reproduce it on my local dev env and I can easily test it by setting ticket_lifetime = 10s on the server's krb5.conf, so I'm happy to test any patch. The client is in Python, based on requests-gssapi. I create a client session, have it make a server call every 2s and surely enough after the 5th call it's getting a 401. I can try to make a minimal reproducing environment if that would help.

[simo5]

The expiration is one of the data points in the encrypted part (MagBearerToken).

Can you detail better how you reproduced? What have you changed, and where?

Are you preforming constrained delegation on the server?

[abompard]

The expiration is one of the data points in the encrypted part (MagBearerToken).

Ah, that makes sense :-) If I understand correctly, the mod_session cookie is shared with other applications on the Apache instance, so it's not really possible to have it expire with the creds, right? Is there a way for me to look into this expiration date in the encrypted token?

Can you detail better how you reproduced? What have you changed, and where?

This is the relevant apache config part:

<LocationMatch "/fasjson/v[0-9]+/">
  AuthType GSSAPI
  AuthName "Kerberos Login"
  GssapiUseSessions On
  Session On
  SessionCookieName fasjson_session path=/fasjson;httponly;secure;
  SessionHeader FASJSONSESSION
  GssapiSessionKey file:/run/fasjson/session.key
  GssapiCredStore keytab:/etc/httpd/conf/fasjson.keytab
  GssapiCredStore client_keytab:/etc/httpd/conf/fasjson.keytab
  GssapiCredStore ccache:FILE:/run/fasjson/krb5ccache
  GssapiImpersonate On
  GssapiDelegCcacheDir /run/fasjson/ccaches
  GssapiDelegCcachePerms mode:0660
  GssapiUseS4U2Proxy on
  GssapiAllowedMech krb5

  Require valid-user
</LocationMatch>

I've also set ticket_lifetime = 10s in the [libdefaults] section of /etc/krb5.conf

This is the client:

from time import sleep
from fasjson_client import Client
c = Client("https://fasjson.tinystage.test/fasjson/")
for i in range(10):
    print(c.whoami())  # This makes the authenticated HTTP call
    sleep(2)

The client class initialization code is here: https://github.com/fedora-infra/fasjson-client/blob/dev/fasjson_client/gss_http.py

Are you preforming constrained delegation on the server?

Yes! It's the delegated credentials that expire before the mod_session's session.

[simo5]

Ok, it seems to me the problem is that you have a shorter ticket_lifetime on the server than on clients, therefore when the server take a delegated ticket it will have a shorter expiration time even if the client ticket is still valid.

A simple way to fix this is to make the lifetime on the server as long as a the longest ticket lifetime on any client (I expect 24h to be a common value).

Other than that I need to think if it is possible to better handle this on the server side. Technically nothing should complain about an expired ticket because the server should simply be able to use the original client ticket (if the hypothesis that it is valid for longer holds true) to obtain new fresh credentials ...

[abompard]

On the actual servers the ticket_lifetime is 24h for each, I've set it short on my testing env because I didn't want to wait a day to see if my attempts to workaround this were successful. But on the actual server and client it's 24h. As I wrote in https://github.com/gssapi/mod_auth_gssapi/issues/316#issuecomment-2357682337 the client ticket was still valid for a long time when the delegated credentials expired on the server.

[simo5]

Can you klist /run/fasjson/krb5ccache as well as the user specific ccache ? I suspect the lifetiem of the ticket you obtain in constrained delegation is clamped to the shorter lifetime between the user ticket and the server TGT used to perform the delegation, so if the server had a ticket in cache obtained hours earlier I can see how the delegated credential could have a shorter lifetime too.

I am not sure yet how to handle this situation, ut I want to figure out the cause before I can suggest a) a workaround and then b) a solution I can implement in actual code

[abompard]

I think you may be right, because I've seen the delegated credentials lifetime be shorter than 24h even right after they were obtained. This is what I currently have on the test server with ticket_lifetime = 10s:

# klist -c /run/fasjson/krb5ccache 
Ticket cache: FILE:/run/fasjson/krb5ccache
Default principal: HTTP/fasjson.tinystage.test@TINYSTAGE.TEST
Valid starting       Expires              Service principal
10/04/2024 09:34:57  10/04/2024 09:35:07  krbtgt/TINYSTAGE.TEST@TINYSTAGE.TEST

# klist -c /run/fasjson/ccaches/admin@TINYSTAGE.TEST 
Ticket cache: FILE:/run/fasjson/ccaches/admin@TINYSTAGE.TEST
Default principal: admin@TINYSTAGE.TEST
Valid starting       Expires              Service principal
10/04/2024 09:34:57  10/04/2024 09:35:07  krbtgt/TINYSTAGE.TEST@TINYSTAGE.TEST
        for client HTTP/fasjson.tinystage.test@TINYSTAGE.TEST
10/04/2024 09:34:20  10/05/2024 08:59:21  HTTP/fasjson.tinystage.test@TINYSTAGE.TEST
        renew until 10/11/2024 09:34:14
10/04/2024 09:34:57  10/04/2024 09:35:07  ldap/ipa.tinystage.test@
        Ticket server: ldap/ipa.tinystage.test@TINYSTAGE.TEST

and this is on the client:

$ TZ=UTC klist
Ticket cache: KCM:1000:63978
Default principal: admin@TINYSTAGE.TEST

Valid starting       Expires              Service principal
04/10/2024 09:34:16  05/10/2024 08:59:21  krbtgt/TINYSTAGE.TEST@TINYSTAGE.TEST
        renew until 11/10/2024 09:34:14
04/10/2024 09:34:20  05/10/2024 08:59:21  HTTP/fasjson.tinystage.test@
        renew until 11/10/2024 09:34:14
        Ticket server: HTTP/fasjson.tinystage.test@TINYSTAGE.TEST

So the client did obtain a ticket for HTTP/fasjson.tinystage.test that is much longer-lived than the delegated credential on the server.

[simo5]

Ok, so I am surprise that the client would even get back a 401 in this scenario, as the delegate credentials are not something mod_auth_gssapi would check during authentication, is your application itself returning the 401 to the client after a failure to use those credentials ?

[abompard]

Ok, so I am surprise that the client would even get back a 401 in this scenario, as the delegate credentials are not something mod_auth_gssapi would check during authentication, is your application itself returning the 401 to the client after a failure to use those credentials ?

Yes, my server checks the lifetime of the delegated credentials and returns 401 if they are expired.

[simo5]

I have been thinking about this for a while, and I do not see a very clean solution yet, I think one way would be for you to remove the expired ccache, drop the session cookie and set the negotiate headers in the 401 response. This will force the client to try to re-authenticate and mod_auth_gssapi will re-create the needed tickets. All three steps will be necessary to be able to obtain a new valid ccache. I am not sure this is something I can easily check in mod_auth_gssapi itself, simply because mod_auth_gssapi may not have access at all to the caches beeing outputted by gssproxy which is generally used for privilege separation and hides these from the "forntend" process.

[simo5]

(edited the comment above as I fat fingered a send mid-typing :)

[abompard]

I have been thinking about this for a while, and I do not see a very clean solution

Thanks for looking at it!

I think one way would be for you to remove the expired ccache, drop the session cookie and set the negotiate headers in the 401 response.

OK, can the application behind mod_auth_gssapi set the Negotiate headers? I thought only mod_auth_gssapi would be able to do it. Is there a way to ask it to do so? Maybe via a response header like what mod_session does?

I am not sure this is something I can easily check in mod_auth_gssapi itself, simply because mod_auth_gssapi may not have access at all to the caches beeing outputted by gssproxy which is generally used for privilege separation and hides these from the "forntend" process.

Understood. It would work if the application behind mod_auth_gssapi was able to ask it to set the Negotiate headers, then, no?

[simo5]

You cn easily set the Negotiate headers, the first header is quite standard and is literally just: WWW-Authenticate: Negotiate

[abompard]

You cn easily set the Negotiate headers, the first header is quite standard and is literally just: WWW-Authenticate: Negotiate

Oh! I didn't realize that :-) I just tried that instead of the redirect+set-cookie, and it worked great! Even without removing the delegated ccache or clearing the cookie. For the latter I suppose it's because the client http library ignores the cookie when the negotiate header is set, so I'd rather not rely on that and I'll clear it anyway. But this solution is much much better than the redirect, because I don't have to update the client code :-) (and it's also one less round-trip) Thanks!

[simo5]

Great news!