search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
8.83k stars 1.62k forks source link

[BUG] `_restore` api throwing 500 error on security enabled 3.0.0 cluster #13548

Open rishabh6788 opened 1 week ago

rishabh6788 commented 1 week ago

Describe the bug

Our nightly benchmark runs for 3.0.0 clusters are failing for snapshot enabled runs. While running _restore for snapshot after registering the repository we are seeing 500 error below log:

[2024-05-05T18:41:05,704][WARN ][r.suppressed             ] [ip-172-31-169-113.ec2.internal] path: /_snapshot/benchmark-workloads-repo-300/http_logs_default/
_restore, params: {repository=benchmark-workloads-repo-300, snapshot=http_logs_default}
org.opensearch.transport.RemoteTransportException: [seed][172.31.144.199:9300][cluster:admin/snapshot/restore]
Caused by: org.opensearch.OpenSearchSecurityException: Unexpected exception cluster:admin/snapshot/restore
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:479) ~[?:?]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:133) ~[opensearch-3.0.0.jar:3.0.
0]
        at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:129) ~[opensearch-3.0.0.jar:3.0.
0]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:207) ~[?:?]
        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:316) ~[?:?]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:155) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:844) ~[?:?]
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:116) ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:272) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:140) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:120) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:108) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:100) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:772) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.forwardFragments(NativeInboundBytesHandler.java:156) [opensearch-3.0.0.jar:3.0.0
]
        at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.doHandleBytes(NativeInboundBytesHandler.java:93) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:143) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:119) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-3.0.0.jar:3.
0.0]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.109.Final.jar:4.1.10
9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.10
9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.
Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.109.Final.jar:4.1.1
09.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.109.Final.jar:4.1.1
09.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.109.Final.jar:4.1.109
.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.109.Final.jar:4.1.10
9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.10
9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.
Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) [netty-handler-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) [netty-handler-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) [netty-handler-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) [netty-codec-4.1.109.Final.jar:4.1.109.F
inal]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.109.Final.jar:4.1.10
9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.10
9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.
Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.109.Final.jar:4.1.109.Final
]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.109.Final.jar:4.1.10
9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.10
9.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.109.Final.jar:4.1.109.Final]
        at java.lang.Thread.run(Thread.java:1583) [?:?]

Seems to be related to https://github.com/opensearch-project/OpenSearch/pull/8922.

Related component

Storage:Snapshots

To Reproduce

  1. Create a security enabled 3.0.0 cluster
  2. Register snapshot repo and take a snapshot
  3. Restore the snapshot.

Expected behavior

The _restore api should return 200 but it returns 500 instead.

Additional Details

No response

andrross commented 1 week ago

[Triage - attendees 1 2 3 4] @rishabh6788 Thanks for filing, it does indeed look like a bug. Is this a remote store enabled cluster? Issue #8922 was for the remote store restore but the repro steps here do not mention remote store.