Trying to connect opensearch link but getting error - 502 Bad Gateway nginx.

[Rajeshdaneti]

Hi All, I am trying to connect Opensearch 2.2.0/2.3.0 database with link - https://opensearch.opensearch-import-test.localhost.cloud but getting error - 502 Bad Gateway. Logs shows following errors- [opensearch-cluster-master-0] path: /bad-request, params: {} java.lang.IllegalArgumentException: invalid version format: ? 8À,À0 ?̨̩̪À+À/ ?À$À( KÀ#À' GÀ at io.netty.handler.codec.http.HttpVersion.(HttpVersion.java:116) ~[netty-codec-http-4.1.79.Final.jar:4.1.79.Final] at io.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:78) ~[netty-codec-http-4.1.79.Final.jar:4.1.79.Final]

[2023-01-06T09:49:01,998][DEBUG][r.suppressed ] [opensearch-cluster-master-0] path: /bad-request, params: {} java.lang.IllegalArgumentException: text is empty (possibly HTTP/0.9) at io.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:65) ~[netty-codec-http-4.1.79.Final.jar:4.1.79.Final] at io.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:124) ~[netty-codec-http-4.1.79.Final.jar:4.1.79.Final]

It is working fine when trying in version 1.3.2. Problem is only with opensearch 2.2.0/2.3.0. Please help to resolve the issue.

Regards Rajesh Kumar.

[dblock]

Try with curl, what do you see? Is SSL enabled? How do you deploy the instance? Can you connect to the instance locally first?

[Rajeshdaneti]

Hi Daniel,

Thank you very much for your support. Please find the answers below - 1) Curl also getting the same error- Tried below commands curl https://opensearch.opensearch-import-test.xxx.xxxxxxxxx.cloud/ curl XGET https://opensearch.opensearch-import-test.xxx.xxxxxxxxx.cloud/

502 Bad Gateway

---

nginx

2)SSL is not enabled because we set network.host: 0.0.0.0 plugins.security.disabled: true

3)We deploy the instance through HELM charts and kubernates on cloud.

4)Connecting in local instance. curl -XGET http://localhost:9200 { "name" : "opensearch-cluster-master-0", "cluster_name" : "opensearch-cluster", "cluster_uuid" : "wgQfy7CySAWlzrLCNwQ1TA", "version" : { "distribution" : "opensearch", "number" : "2.2.0", "build_type" : "tar", "build_hash" : "b1017fa3b9a1c781d4f34ecee411e0cdf930a515", "build_date" : "2022-08-09T02:27:25.256769336Z", "build_snapshot" : false, "lucene_version" : "9.3.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } Logged into pod and run above command.

Please help me to resolve the issue.

Regards Rajesh Kumar

[Rajeshdaneti]

Hi Daniel, Any update on the issue. Please help me to resolve the issue because it is blocker for me.

Regards Rajesh Kumar

[peterzhuamazon]

Hi @Rajeshdaneti let me know if this is still a blocker. This seems like an issue for the core, transfer. Thanks.

[Rajeshdaneti]

Hi Peter, Thankyou very much for your reply.

Even after setting up ingress, I am still having the same problem. We are using "-XPUT -H 'Content-Type: application/json' http://opensearch-cluster-master:9200" as a workaround, but this only functions within the kubernetes namespace pod. The link https://opensearch.opensearch-import-test.xxx.xxxxxxxxx.cloud/ should function from any where like web browser, shell script etc within a secured network in accordance with our requirement based on namespace and environment setup.

Please help me to resolve the issue.

Regards Rajesh Kumar.

[peterzhuamazon]

Hi @Rajeshdaneti like @dblock have mentioned, we have no idea how you deploy the cluster. I have no context on what exactly is behind https://opensearch.opensearch-import-test.xxx.xxxxxxxxx.cloud/ so there is really not much I can suggest here.

If you can describe how you deploy the instance it would be great help. We also have a helm-charts repo in opensearch-project/helm-charts if you want to deploy on kubernetes clusters.

Thanks.

[Rajeshdaneti]

Hi Peter, Yes, we are installing opensearch and opensearch dashboards through helm charts.

Requirement - After deploying OpenSearch via the ci/cd process, we import stored report data from one environment to another, create shards with specific characteristics like refresh interval, and import exported dashboards using NDJSON. Also, we are attempting to generate the cluster/node health status via a web browser. The link http://opensearch-cluster-master:9200/ works properly when we install OpenSearch with other applications like microservices, kafka, etc. in the same namespace because they are in the same namespace. However, when importing dashboard/saved report data for different namespaces (here, opensearch is referred to as a centralized/standalone namespace) through ci/cd process does not work in the most recent versions 2.2.0/2.3.0/2.5.0 but it was working in version 1.3.2. We are deriving the dynamic link "https://opensearch.$namespace.$cluster.$dnszone.cloud/" through config variables in ci/cd process.

Please suggest.

Regards Rajesh Kumar.

[zalseryani]

Kindly any update on this issue ? I am facing the same case when trying to access opensearch using a domain routed to ingress on kubernetes cluster. Its very weird that no one is checking this case ! Thank you

[dblock]

I'm going to move this back to opensearch-devops.

@Rajeshdaneti are you saying your exact setup works with 1.x but not 2.x?

@zalseryani what is the problem you're having and what's the easiest repro that anyone could look at?

Both: the original error in the bug report 502 Bad Gateway, logs shows following errors:

[opensearch-cluster-master-0] path: /bad-request, params: {}
java.lang.IllegalArgumentException: invalid version format: ? 8À,À0 ?̨̩̪À+À/ ?À$À( KÀ#À' GÀ

Tells me that there's an SSL request to the host while the host doesn't have SSL enabled. Possible?

[dblock]

I think I know what's going on. By using plugins.security.disabled you are disabling the security plugin, not TLS. See https://opensearch.org/docs/latest/security/configuration/tls/ for details of how to configure TLS. To disable it, use plugins.security.ssl.http.enabled. There's also https://stackoverflow.com/questions/74003985/opensearch-disable-ssl-https-but-not-securityplugin that talks about it.

Does adding plugins.security.ssl.http.enabled help @zalseryani?

[zalseryani]

I will test and keep you updated.

thank you

[jordarlu]

Greetings, @zalseryani ,just want to follow up on this issue.. Have you got a chance to test with what @dblock mentioned above?
thanks,

[zalseryani]

@jordarlu

Yes it worked after using http.enabled=false

Thank you @dblock for your time and support.

[LHozzan]

Hi guys.

Today is second time, where I spot same problem on our infrastructure. Unfortunately, the problem occurring random. Actually I dont see any pattern, I am not able to replica it.

Error messages looks like:

main {"type": "logging", "timestamp": "2023-06-27T07:09:48,568Z", "level": "ERROR", "component": "o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport", "cluster.name": "logging", "node.name": "opsr-data-0", "message": "Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)", "cluster.uuid": "6WbiZY0ZQFyBc9salzNFJQ", "node.id": "FdhE_dzcRaaUxL7GjnKAag" , 
main "stacktrace": ["javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]",
main "at sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]",
main "at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]",
main "at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
main "at java.lang.Thread.run(Thread.java:833) [?:?]",
main "Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1898) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159) ~[?:?]",
main "at sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]",
main "... 27 more"] }

main {"type": "logging", "timestamp": "2023-06-27T07:09:48,569Z", "level": "WARN", "component": "o.o.h.AbstractHttpServerTransport", "cluster.name": "logging", "node.name": "opsr-data-0", "message": "caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/100.96.14.9:9200, remoteAddress=/100.96.15.55:38712}", "cluster.uuid": "6WbiZY0ZQFyBc9salzNFJQ", "node.id": "FdhE_dzcRaaUxL7GjnKAag" , 
main "stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
main "at java.lang.Thread.run(Thread.java:833) [?:?]",
main "Caused by: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]",
main "at sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]",
main "at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]",
main "at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "... 16 more",
main "Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1898) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159) ~[?:?]",
main "at sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]",
main "at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]",
main "at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "... 16 more"] }
main "at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
main "at java.lang.Thread.run(Thread.java:833) [?:?]",
main "Caused by: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]",
main "at sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]",
main "at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]",
main "at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "... 16 more",
main "Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1898) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159) ~[?:?]",
main "at sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]",
main "at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]",
main "at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "... 16 more"] }

main {"type": "logging", "timestamp": "2023-06-27T07:09:48,912Z", "level": "WARN", "component": "o.o.h.AbstractHttpServerTransport", "cluster.name": "logging", "node.name": "opsr-data-0", "message": "caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/100.96.14.9:9200, remoteAddress=/100.96.16.47:38792}", "cluster.uuid": "6WbiZY0ZQFyBc9salzNFJQ", "node.id": "FdhE_dzcRaaUxL7GjnKAag" , 
main "stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
main "at java.lang.Thread.run(Thread.java:833) [?:?]",
main "Caused by: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]",
main "at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]",
main "at sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]",
main "at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]",
main "at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "... 16 more",
main "Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)",
main "at sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1898) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196) ~[?:?]",
main "at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159) ~[?:?]",
main "at sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]",
main "at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]",
main "at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]",
main "at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "... 16 more"] }
main "at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]",
main "... 16 more"] }

As you can see, many times is repeating one Java error message: main "Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)". Have we something set wrong?

We using OpenSearch version 2.7.0 in stable setup. I mean, after node start/restart (in the case it is only one node with all roles, but I spoted the problem in bigger cluster, where we using more nodes with separated roles) everything looks good and everything working as we expecting. After several time (ones per week, ones per month, ... just random occurrence) we get alert about cluster inaccessible, due to server malfunction (HTTP 502). As a clients we using FluentBits 2.1.4, which when problem with OpenStart begin, start logging warnings:

[2023/06/26 20:36:15] [ warn] [engine] failed to flush chunk '28-1687811736.549959943.flb', retry in 67 seconds: task_id=199, input=tail.0 > output=opensearch.2 (out_id=2)
[2023/06/26 20:36:15] [ warn] [engine] failed to flush chunk '28-1687811774.548822940.flb', retry in 6 seconds: task_id=243, input=tail.0 > output=opensearch.2 (out_id=2)
[2023/06/26 20:36:15] [ warn] [engine] failed to flush chunk '28-1687811774.543593141.flb', retry in 10 seconds: task_id=231, input=tail.0 > output=opensearch.2 (out_id=2)
[2023/06/26 20:36:15] [error] [tls] error: unexpected EOF
[2023/06/26 20:36:15] [error] [tls] error: unexpected EOF
[2023/06/26 20:36:15] [error] [tls] error: unexpected EOF

After OpenSearch node restart, FluentBits continue with their tasks without any problems:

[2023/06/27 07:40:52] [ info] [engine] flush chunk '650-1687849401.542287065.flb' succeeded at retry 9: task_id=107, input=tail.0 > output=opensearch.2 (out_id=2)
[2023/06/27 07:38:57] [ info] [engine] flush chunk '659-1687849297.901730481.flb' succeeded at retry 9: task_id=17, input=tail.0 > output=opensearch.2 (out_id=2)
[2023/06/27 07:37:14] [ info] [engine] flush chunk '965-1687849423.197511273.flb' succeeded at retry 9: task_id=8, input=kmsg.2 > output=opensearch.1 (out_id=1)

Huge problem for us is, that when the OpenSearch start rejecting connection over night, we losing valuable data (FluentBits attempts delivering data, but after several failed, they drop them).

In the case, that you need some additional information, please, let me know.

Best regards.

[jordarlu]

Sorry for the late response, @LHozzan. As we are tring to avoid cascading down multiple cases in a single issue which may cause difficulty in tracking afterwards, I hope you don't mind to open a new issue for this problem in https://github.com/opensearch-project/opensearch-devops. We would love to dig into it with you...

By searching for the error content, would you check this post out and see if it helps by any chance? thanks !!

[LHozzan]

Hi @jordarlu .

Thank you for feeedback.

Meantime we upgraded OpenSearch in our logging stack to version 2.8 and actually we have (one week so far) 2.9. We also upgraded client, FluentBit to 2.1.8 version. After upgrade to v2.8 it seems, that the problem disappeared. We not spot the problem anymore, but as I mentioned early, it happening random on random clusters (= random hyperscalers).

If you agree, let us test v2.9 in our clusters and when we will preparing to upgrade to higher version (2.10?) and without the problem, I think we can consider it as a solved, due to placing fixes in the Java RE. What do you think?

[gaiksaya]

Closing this issue based on https://github.com/opensearch-project/opensearch-devops/issues/120#issuecomment-1701516962 please re-open if issue still exists. Thanks!

[gokul3062]

Hello All , I am using AWS opensearch service for dev environment I am using t3.small.search instance type and 1 node.

Some time due to some process , Do no exactly know what happens there show 502 gateway error

1. Need to know when does this error occurs
2. If somekind of this error occurs, can we restart service in AWS is it possible?