Lots of 401 requests + read/seek syscalls observed

[dky]

Hello guys, Thank you for your work on this project it's super useful!

I'm writing because we are currently running this in production and our devs have recently started to complain about Nginx performing poorly. After I started digging into the issue I observed a lot of requests were throwing 401s in the access logs. Users are getting authed however it looks like every subsequent request turns out to be a 401. It appears like every visitor is failing auth then authed. More below.

We dug further and when things get busy we noticed Nginx doing a lot of I/O. Roughly 1MB/s at times per worker thread. At first, we thought it was logging so we disabled all Nginx logging however the I/O remained. I ran strace on the Nginx worker thread and found thousands upon thousands of read and lseek requests for what appears for kerb tickets for users. I'll see the SPN + a read call with a SHA:256 followed by a hash + the user ID of the user. The thing is there are duplicates for the user almost as if these calls aren't being cached somehow and I suspect this is what is causing a majority of the I/O.

Do you guys have any hints on what this behavior may be and any tips on debugging?

Thanks!

[jheiss]

The behavior of the client sending a request, getting a 401 (with a WWW-Authenticate header indicating that authentication is required), and then resending the request with authentication is the standard behavior of HTTP authentication. One way to mitigate that is to configure clients to preemptively include the Kerberos authentication on their requests. For example, with Python and requests-kerberos you can pass force_preemptive=True.

[dky]

Thanks for the reply. In this case, I'd expect to see 1 401 requests per session however, I'm seeing nearly half of the visitor's traffic as 401. It seems as if every subsequent HTTP request is doing some type of 401 re-auth.

[jheiss]

Unless you've done something special to bypass authentication on subsequent requests, every single HTTP request will go through the 401 double round trip.

You mentioned a session. Are you setting a session cookie and expecting that to serve to authenticate the user? We do that in some services where we use application-level implementations of Negotiate authentication. This nginx plugin does not know about your session cookies though, so it's going to require authentication before the request gets to your application and the session cookie is inspected.

[dky]

Ah, Thanks for this! I did not know about "Unless you've done something special to bypass authentication on subsequent requests, every single HTTP request will go through the 401 double round trip." This would explain why I'm seeing a 401 for every single request leading to the 50% of 401/200 traffic split. We are not doing any special sessions, I was using that as a reference for a visitor session. The session cookie does sound nice and to me would eliminate all the additional 401's for every request. I guess we'll dig into that.

So because every single request is getting re-authenticated that would explain why we could be potentially seeing the Nginx syscalls to perform the kerb ticket verification.