[BUG] java.lang.NoSuchMethodError in a REST handler causes a OpenSearchSecurityException

[dblock]

What is the bug?

I installed an incompatible plugin into OpenSearch that has a REST handler. That handler raises a NoMethodError in this particular instance of OpenSearch, causing the following API response.

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "Unexpected exception api"
      }
    ],
    "type": "security_exception",
    "reason": "Unexpected exception api"
  },
  "status": 500
}

The root cause is a NoSuchMethodError in the plugin.

[2024-01-11T19:48:17,964][ERROR][o.o.s.f.SecurityFilter   ] [b6e284bbf0cb] Unexpected exception java.lang.NoSuchMethodError: 'org.opensearch.core.xcontent.XContentBuilder org.opensearch.rest.RestController.toXContent(org.opensearch.core.xcontent.XContentBuilder, org.opensearch.core.xcontent.ToXContent$Params)'
java.lang.NoSuchMethodError: 'org.opensearch.core.xcontent.XContentBuilder org.opensearch.rest.RestController.toXContent(org.opensearch.core.xcontent.XContentBuilder, org.opensearch.core.xcontent.ToXContent$Params)'
    at org.opensearch.plugin.api.action.APIResponse.toXContent(APIResponse.java:74) ~[?:?]
    at org.opensearch.plugin.api.RestAPIAction$1.buildResponse(RestAPIAction.java:72) ~[?:?]
    at org.opensearch.plugin.api.RestAPIAction$1.buildResponse(RestAPIAction.java:68) ~[?:?]
    at org.opensearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:53) ~[opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:52) ~[opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:64) ~[opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) ~[opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) ~[opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.plugin.api.action.TransportAPIAction.doExecute(TransportAPIAction.java:30) ~[?:?]
    at org.opensearch.plugin.api.action.TransportAPIAction.doExecute(TransportAPIAction.java:21) ~[?:?]

But this causes

[2024-01-11T20:09:49,425][WARN ][r.suppressed             ] [a0fd2fdfc7e0] path: /_plugins/api, params: {}
org.opensearch.OpenSearchSecurityException: Unexpected exception api
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:479) [opensearch-security-2.11.1.0.jar:2.11.1.0]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.11.1.0.jar:2.11.1.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.plugin.api.RestAPIAction.lambda$prepareRequest$0(RestAPIAction.java:68) [api-2.11.1.jar:2.11.1]
    at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.security.filter.SecurityRestFilter.lambda$wrap$2(SecurityRestFilter.java:187) [opensearch-security-2.11.1.0.jar:2.11.1.0]
    at org.opensearch.rest.RestController.dispatchRequest(RestController.java:323) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:414) [opensearch-2.11.1.jar:2.11.1]
    at org.opensearch.rest.RestController.dispatchRequest(RestController.java:252) [opensearch-2.11.1.jar:2.11.1]

Reproduce

1. OpenSearch 2.11 standard docker.
2. Build and install opensearch-api from https://github.com/dblock/OpenSearch/tree/2.11.1-api-plugin with ./gradlew :plugins:api:assemble.
3. curl -u admin:admin https://localhost:9200/_plugins/api

What is the expected behavior?

An internal server error, not a security_exception.

Do you have any additional context?

Installing a plugin built from https://github.com/dblock/OpenSearch/tree/2.11.1-api-plugin.

[peternied]

Thanks for filing

The underlying issue is that after permissions checks occur, chain.proceed(...) [1] is called to execute the task, this code is inside a try block to ensure any exceptions in privilege evaluation are not leaked to callers. Exceptions generated from the task itself should not be swallowed captured by this catch block, but passed up the exception stack.

• [1] https://github.com/opensearch-project/security/blob/2.11/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L395

[stephen-crawford]

[Triage] Hi @dblock, thanks for filing this issue. We can move forward when either the exception type is fixed (short term) or a more extensive correction is done following Peter's comment.