Metrics not going to ES5.x

[gopi-nath]

Hi,

I am not able to send data to es5.x

Following is my stagemonitor.properties stagemonitor.applicationName=name stagemonitor.instrument.include=com.evive.name stagemonitor.init.async=true stagemonitor.active=true stagemonitor.reporting.interval.elasticsearch=1 stagemonitor.elasticsearch.numberOfReplicas=1 stagemonitor.elasticsearch.numberOfShards=100 stagemonitor.instrument.runtimeAttach=true stagemonitor.elasticsearch.availabilityCheckPeriodSec=5 stagemonitor.elasticsearch.url=https://user:pass@data3:9202/ stagemonitor.web.collectPageLoadTimesPerRequest=true stagemonitor.plugins.disabled=JdbcPlugin stagemonitor.internal.monitoring=true

While starting the application, I get the following logs and then no logs regarding stagemonitor,(no errors or warns).

19 May 2017 10:25:41,815 INFO [main] Application:57 - [] - [] - Started Application in 28.556 seconds (JVM running for 87.871) 19 May 2017 10:25:59,925 INFO [stagemonitor-startup] Stagemonitor:98 - [] - [] - Measurement Session is initialized: [application=Admin] [instance=localhost] [host=mu] 19 May 2017 10:25:59,928 INFO [stagemonitor-startup] Stagemonitor:142 - [] - [] - Initializing plugin WebPlugin 19 May 2017 10:26:00,008 INFO [stagemonitor-startup] Stagemonitor:142 - [] - [] - Initializing plugin RequestMonitorPlugin 19 May 2017 10:26:00,051 INFO [stagemonitor-startup] Stagemonitor:142 - [] - [] - Initializing plugin CorePlugin 19 May 2017 10:26:00,082 INFO [stagemonitor-startup] CorePlugin:557 - [] - [] - Not sending metrics to InfluxDB (url=null, interval=60s) 19 May 2017 10:26:00,083 INFO [stagemonitor-startup] CorePlugin:565 - [] - [] - Sending metrics to Elasticsearch ([https://user:pass@10.13.1.35:9202/]) every 1s 19 May 2017 10:26:00,152 INFO [stagemonitor-startup] Stagemonitor:142 - [] - [] - Initializing plugin JvmPlugin 19 May 2017 10:26:00,238 INFO [stagemonitor-startup] Stagemonitor:134 - [] - [] - Not initializing disabled plugin JdbcPlugin

When checked ES logs, found the following exception:

[2017-05-19T10:24:09,624][WARN ][c.e.p.a.t.Netty4TLSHttpServerTransport] [Stuart] caught exception while handling client http traffic, closing connection [id: 0xbc1ba37f, L:0.0.0.0/0.0.0.0:9202 ! R:/10.14.1.94:38200] io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:721) ~[?:?] at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:954) ~[?:?] at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:908) ~[?:?] at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:985) ~[?:?] at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1028) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1028) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:950) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[?:?] ... 15 more

How do i provide SSL support here?

[trampi]

Hi there! To send to a elasticsearch backend that is secured via https, you should add the certificate to the trust store of the application sending the metrics.

The trust store is a place for trusted root certificates. The following article provides an introduction on how to import the certificate in to the trust store: https://connect2id.com/blog/importing-ca-root-cert-into-jvm-trust-store Edit to clarify: you should import the certificate of the elasticsearch host. Look here to find out how to extract the certificate: https://superuser.com/questions/97201/how-to-save-a-remote-server-ssl-certificate-locally-as-a-file

The alternative, if you do not want to modify the trust store, is to obtain the certificate from a trusted certificate provider, like Let's Encrypt. Let's Encrypt issues free certificate, but your system has to be reachable from the internet for the certificate issuing process.

Reach out if you have more questions.

Best regards, Fabian

[gopi-nath]

Hi,

Thanks for the help. I already tried that. I added the certificate to the trust store. But i'm still getting the same exception.

[jay-dihenkar]

Is there any way to support mutual SSL while sending index request to ES? We have a setup that requires mutual SSL so the client will need client cert + req.

[felixbarny]

@jay-dihenkar I have not tried this yet. You could try following the suggestions in this stack overflow thread: https://stackoverflow.com/a/2240993/1125055

@gopinath-evive How did you configure elasticsearch to use ssl encryption? Are you using x-pack security?

Also the config stagemonitor.elasticsearch.numberOfShards=100 seems a bit high. Why do you want that many shards?

[gopi-nath]

@felixbarny numberOfShards=100 was just for testing purpose. I have actually set it to numberOfShards=1.

The issue is resolved. It was a certificate issue on the server.

Also could you please let me know when https://github.com/stagemonitor/stagemonitor/commit/7c41e3759fe3ebc2738564506e05411c58b256b0 (ElasticsearchClient.java) will be released ?

I'm getting null pointer exception in 0.31.0.

[felixbarny]

@gopi-nath 0.80.0.RC1 will be released within the next few days.

How did you resolve the issue on your server?

[gopi-nath]

@felixbarny : We run a mutual SSL setup for ES <--> Application. Now I didn't find any way to setup the mutual ssl via the http client used in the stagemonitor. So, we used a reverse proxy infront of ES to handle mutual SSL and let stagemonitor talk to it.

[felixbarny]

Thx, that's good to know. In the future, I might use the official Elastichsearch REST client.