Oathkeeper returns a 401 "Access credentials are invalid" when exceeding Ory Network's rate limiting on /sessions/whoami endpoint

wewelll commented 4 months ago

Preflight checklist

[X] I could not find a solution in the existing issues, docs, nor discussions.
[X] I agree to follow this project's Code of Conduct.
[X] I have read and am following this repository's Contribution Guidelines.
[X] I have joined the Ory Community Slack.
[X] I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

We're using Ory Network as our authentication provider, and Oathkeeper to protect our APIs. Oathkeeper is configured with a cookie_session authenticator and uses the /sessions/whoami endpoint to check the cookie.

When a user makes a lot of calls to our API at the same time, Oathkeeper can sometimes return a 401 - Access credentials are invalid response, even when the cookie is valid.

I did a bit of digging with our OpenTelemetry instrumentation, and figured out that the /sessions/whoami endpoint is in fact returning a 429 status, because we're exceeding rate limiting.

I see two potential improvements:

Change the response status

Instead of returning a misleading 401 status, Oathkeeper should return another status code, like 429, so it can help the end-user understand what is happening.

Add cache to the check_session_url call

When a user performs a lot of calls in a small timeframe, it's not necessary to call the /sessions/whoami for all calls because the session cookie hasn't changed. I suggest to add an optional caching mechanism to the cookie_session authenticator, it would solve this issue even with a low TTL (~1s) This caching mechanism could be similar to the one found in the oauth2_client_credentials authenticator.

Reproducing the bug

Run Oathkeeper with a cookie_session authenticator linked to an Ory Network project.
Log-in and retrieve the session cookie
Write and run a script that sends 100 requests in parallel to the running Oathkeeper, with the session cookie
You should see a 401 at some point, even though the cookie is valid

Relevant log output

No response

Relevant configuration

authenticators:
  cookie_session:
    enabled: true
    config:
      check_session_url: https://my-project-slug.projects.oryapis.com/sessions/whoami
      preserve_path: true
      extra_from: '@this'
      subject_from: 'identity.id'

Version

latest

On which operating system are you observing this issue?

macOS

In which environment are you deploying?

Kubernetes

Additional Context

No response

aeneasr commented 4 months ago

That makes sense, we should definitely return 429 in this scenario instead of 401 which is masking the underlying reason for the error.

wewelll commented 2 months ago

This issue also shows that when we build an API on top of Oathkeeper & Ory Network, the rate-limiting of our API can't be better than Ory Network's rate-limiting on the whoami endpoint.

This is quite a big limitation ... I was suggesting to add a cache mechanism based on the session cookie, in order to avoid unnecessary requests to the whoami endpoint. If the TTL is low (~1s for instance), the security trade-off is acceptable.

Do you think that it would make sense to add this feature ? I see that it exists for other authenticators like oauth2_client_credentials.

In our case, we are trying to lift this limitation by creating our own whoami that has cache, but IMO it would be simpler to have it directly in oathkeeper

ory / oathkeeper