opensearch-project / security

🔐 Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields
https://opensearch.org/docs/latest/security-plugin/index/
Apache License 2.0
188 stars 272 forks source link

[BUG] Critical Security Admin APIs shouldn't trip memory circuit breakers #4687

Open Bukhtawar opened 3 weeks ago

Bukhtawar commented 3 weeks ago

What is the bug?

The below trace was observed on a cluster operating at 95% JVMMP and most likely the API request got tripped by memory circuit breaker

'429 Client Error: Too Many Requests for '
'url:  https://<endpoint>/_opendistro/_security/api/securityconfig']

On close evaluation, I noticed that that API actions for security configs don't override this method defaulting to true

https://github.com/opensearch-project/OpenSearch/blob/c71060e29f56918b87a78237ff117d9018456428/server/src/main/java/org/opensearch/rest/RestHandler.java#L62-L64

    default boolean canTripCircuitBreaker() {
        return true;
    }

How can one reproduce the bug? Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

What is the expected behavior? CB shouldn't trip critical security admin API requests

What is your host/environment?

Do you have any screenshots? If applicable, add screenshots to help explain your problem.

Do you have any additional context? Add any other context about the problem.

cwperks commented 1 week ago

[Triage] Thanks for filing this issue @Bukhtawar . I agree that security APIs should not trip the circuit breaker.

iamsharmaapoorv commented 2 days ago

Hey, Interested in picking this up.