Add Open Distro for ES integration

[otisg]

See: https://opendistro.github.io/for-elasticsearch-docs/docs/pa/api/ https://opendistro.github.io/for-elasticsearch-docs/docs/pa/reference/

Note that PA is open-source, so one can look at how it fetches metrics.

[megastef]

I can confirm the existing ELasticsearch integration in Sematext-Agent is working for Open-Distro-for-Elasticsearch. We have to use open-jdk-11 (due to some HTTPS connection issues).

[otisg]

@bsmid does this translate to needing to update some dependency in the agent?

[bsmid]

In this case it appears there is some ssl incompatibility between open-jdk-8 (on which our agent was) and open-jdk-11 (running Open Distro). Possibly something we can't do anything about, but worth a bit of digging.

[bsmid]

Here is the error Stefan encountered yesterday (agent on jdk 8, opendistro on jdk 11.0.1). It seems there are various ssl related bugs in early versions of jdk 11, could be we just run into one of them here (which made it incompatible with jdk 8 agent). We could try upgrading opendistro jdk to latest available and then try to connect jdk 8 agent to it, to test if this particular issue is fixed.

elasticsearch_1  | javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
elasticsearch_1  |  at sun.security.ssl.Alert.createSSLException(Alert.java:128) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.TransportContext.fatal(TransportContext.java:264) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.TransportContext.fatal(TransportContext.java:259) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLTransport.decode(SSLTransport.java:129) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:672) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:627) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:443) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:422) ~[?:?]
elasticsearch_1  |  at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
elasticsearch_1  |  at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
elasticsearch_1  |  at java.lang.Thread.run(Thread.java:834) [?:?]
elasticsearch_1  | Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
elasticsearch_1  |  at sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1852) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160) ~[?:?]
elasticsearch_1  |  at sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) ~[?:?]
elasticsearch_1  |  ... 26 more```

[otisg]

If it's a bug in the jdk 11, I'd ignore and hope jdk 12, which is out now, fixed it.

[bsmid]

They did produce some ssl bug fixes in later jdk 11 versions too. We could test if jdk12 works well with our agent on jdk 8.

[otisg]

I suggest we don't. Even if it doesn't work what can we do if the bug is in the jdk?

[bsmid]

Not much, it would be interesting to know. There is a chance that besides ssl bugs that exist in jdk we are also doing something wrong, if errors persist even in new versions of 11 and in 12, it might tell us we should check our code.

Regardless of all that, we can produce a general recommendation that in case of ssl errors, people should ensure agent and whatever is being monitored are using the same jdk version (since we saw that helps).

[otisg]

@prog8, @radu-gheorghe, @gr0, which metrics from https://opendistro.github.io/for-elasticsearch-docs/docs/pa/reference/ do you think would be the most valuable to ES users (and that Sematext ES monitoring integration doesn't already collect - https://sematext.com/docs/integration/elasticsearch/ )?

[gr0]

I would like to see a few of the stats available in the Open Distro For Elasticsearch, like: like:

• Thread_Blocked_Time
• Segments_Memory
• Terms_Memory
• StoredFields_Memory
• TermVectors_Memory
• Norms_Memory
• Points_Memory
• DocValues_Memory
• IndexWriter_Memory
• Bitset_Memory
• VersionMap_Memory
• Master_PendingQueueSize
• CB_EstimatedSize
• CB_TrippedEvents
• CB_ConfiguredSize

Some of those are also available in the standard Elasticsearch distribution, so I think we could also enrich our ES monitoring by including them.

[bsmid]

I guess it would then make sense to take whatever we can from "standard" ES stats and use OpenDistro stats only for things ES otherwise doesn't expose.

[gr0]

:+1:

[radu-gheorghe]

Since Open Distro was replaced by OpenSearch (and that is done in #47 ), we can close this one.