opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.81k stars 1.83k forks source link

[BUG] OpenSearch 502 Error with Ingress on Kubernetes #7289

Closed zalseryani closed 1 year ago

zalseryani commented 1 year ago

Issue

Deploying OpenSearch 2.6.0 with Helm chart 2.11.3 with nginx-ingress configuration, and I am not able to access opensearch engine using the ingress domain,

image

Opensearch pods logs when hitting the request

2023-04-24T12:56:17,223][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [opensearch-cluster-master-2] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-04-24T12:56:17,225][WARN ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.48.13.54:9200, remoteAddress=/10.48.13.28:41774}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    ... 16 more
[2023-04-24T12:56:17,227][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [opensearch-cluster-master-1] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-04-24T12:56:17,228][WARN ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.48.10.72:9200, remoteAddress=/10.48.13.28:49812}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
[2023-04-24T12:56:17,230][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [opensearch-cluster-master-0] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    ... 16 more
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-04-24T12:56:17,231][WARN ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-0] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.48.11.187:9200, remoteAddress=/10.48.13.28:57208}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d526571756573742d49443a2035323630303832363030326662306163336665356662303730353136373361380d0a582d5265616c2d49503a2038322e3231322e38362e38360d0a582d466f727761726465642d466f723a2038322e3231322e38362e38360d0a582d466f727761726465642d486f73743a206f70656e7365617263682e6465766f70732d6465762d696e672e70726f67726573736f66742e636c6f75640d0a582d466f727761726465642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d466f727761726465642d536368656d653a2068747470730d0a582d536368656d653a2068747470730d0a617574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a757365722d6167656e743a206375726c2f372e38312e300d0a6163636570743a202a2f2a0d0a0d0a
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]

Kindly advise, Thank you.

dblock commented 1 year ago

Looks like a dup of https://github.com/opensearch-project/opensearch-devops/issues/120, I'll close this one.

I'm 99% sure this is caused by making non-TLS requests from the nginx proxy to a TLS-enabled OpenSearch. I believe plugins.security.ssl.http.enabled=false is what you're looking for.

zalseryani commented 1 year ago

I will test it and keep you updated, thank you very much for your time and support :)

zalseryani commented 1 year ago

@dblock it worked thank you very much :+1:

dblock commented 1 year ago

@zalseryani Glad that worked! Care to grok through the documentation and see if we're doing a poor job documenting this? (And maybe contribute some updates if you have time?).

zalseryani commented 1 year ago

@dblock

I would like to provide support in documentation after having a very good experience in administrating opensearch.

Any repo for doing that or having a policy for such a contribution?

Thank you and sorry for the delay.

YeonghyeonKO commented 4 months ago

Is there anyone who deployed via opensearch-operator resolve the same issue? plugins.security.ssl.http.enabled=false option doesn't allow security config enabled. (ex. securityconfig-update Job cannot connect to a cluster, so .opendistro_security index isn't created)

@zalseryani @dblock What was your environment when you configured the opensearch cluster? My host/environment is as below:

Gihchathur commented 3 months ago

@YeonghyeonKO, I believe the issue might be due to the demo configurations. Setting the OPENSEARCH_INITIAL_ADMIN_PASSWORD automatically enables the demo configs, which override your plugins.security.ssl.http.enabled=false setting. You can check the opensearch.yml file for these demo configurations.

YeonghyeonKO commented 3 months ago

@Gihchathur I chose the way that opensearch k8s operator generates TLS certificates for both Transport(9300) and HTTP(9200?).

[opensearch@test-opensearch-cluster-ml-0 config]$ ls
jvm.options    log4j2.properties         opensearch-notifications-core  opensearch-performance-analyzer  opensearch-security  opensearch.yml  tls-transport
jvm.options.d  opensearch-notifications  opensearch-observability       opensearch-reports-scheduler     opensearch.keystore  tls-http
[opensearch@test-opensearch-cluster-ml-0 config]$ cd tls-http
[opensearch@test-opensearch-cluster-ml-0 tls-http]$ ls
ca.crt  tls.crt  tls.key

opensearchCluster.dashboards has a config about ssl.verificationMode so that I can open url(http://test-opensearch-cluster-dashboards.xxx.com) without tls security. But what I want also is an endpoint of cluster itself (for indexing data from outside).. do you have any idea in this situation?

This is my yaml:

opensearchCluster:
  enabled: true
  general:
    httpPort: "9200"
    image: harbor-srep01.xxx.com/library/opensearchproject/opensearch:v2.15.0
    serviceName: "test-opensearch-cluster"
    drainDataNodes: true
    # https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/userguide/main.md#security-context-for-pods-and-containers
    setVMMaxMapCount: true # In some cases, set general.setVMMaxMapCount to false as this feature also launches an init container with root
    podSecurityContext:
      runAsUser: 1000
      runAsGroup: 1000
    securityContext:
      allowPrivilegeEscalation: true
      privileged: true
  # https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/userguide/main.md#deal-with-max-virtual-memory-areas-vmmax_map_count-errors
  # https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/userguide/main.md#custom-init-helper
  initHelper:
    image: "harbor-srep01.xxx.com/nexus/docker-mig/library/busybox:1.31.1"
    imagePullPolicy: IfNotPresent
  dashboards:
    enable: true
    replicas: 1
    image: harbor-srep01.xxx.com/library/opensearchproject/opensearch-dashboards:v2.15.0
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "1Gi"
        cpu: "500m"
    tls:
      enable: false
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    additionalConfig:
      # https://opensearch.org/docs/latest/install-and-configure/install-dashboards/tls/
      opensearch.ssl.verificationMode: none
  nodePools:
    - component: master
      replicas: 3
      pdb:
        enable: false
        # enable: true
        # minAvailable: 1
      diskSize: "10Gi"
      persistence:
        pvc:
          storageClass: "sc-nfs-app-retain"
          accessModes:
           - ReadWriteOnce
      roles:
        - "cluster_manager"
        - "master"
      # https://github.com/opensearch-project/opensearch-k8s-operator/issues/669#issuecomment-1829833573
      # Suggestion: 1000m CPU & 2048Mi memory
      resources:
        requests:
          memory: "4Gi"
          cpu: "1"
        limits:
          memory: "4Gi"
          cpu: "2"
      # env:
      #   - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
      #     value: "hcpOss12~!"
    - component: data
      replicas: 2
      diskSize: "100Gi"
      persistence:
        pvc:
          storageClass: "sc-nfs-app-retain"
          accessModes:
           - ReadWriteOnce
      roles:
        - "data"
        - "ingest"
      resources:
        requests:
          memory: "8Gi"
          cpu: "2"
        limits:
          memory: "8Gi"
          cpu: "4"
    - component: ml
      replicas: 1
      diskSize: "10Gi"
      persistence:
        pvc:
          storageClass: "sc-nfs-app-retain"
          accessModes:
           - ReadWriteOnce
      roles:
        - "ml"
      resources:
        requests:
          memory: "24Gi"
          cpu: "4"
        limits:
          memory: "24Gi"
          cpu: "8"
  security:
    tls:
      transport:
        generate: true
        perNode: true
      # https://opensearch-project.github.io/opensearch-k8s-operator/docs/userguide/main.html#node-httprest-api
      http:
        generate: true
    config:
      adminCredentialsSecret: # these are the admin credentials for the Operator to use
         name: admin-credentials-secret
      securityConfigSecret:  # this is the whole security configuration for OpenSearch
         name: securityconfig-secret

image