search

prometheus-community / elasticsearch_exporter

Elasticsearch stats exporter for Prometheus
Apache License 2.0
1.92k stars 792 forks source link

Cannot connect to Elasticsearch with TLS 1.2 and specific cipher suites #158

Open benno001 opened 6 years ago

benno001 commented 6 years ago

The exporter connection to Elasticsearch fails when Elasticsearch is configured with X-Pack, basic auth, TLS 1.2 and the following xpack.ssl.cipher_suites:

Error messages: <time> elastic elasticsearch_exporter: level=warn ts=<time> caller=nodes.go:1027 msg="failed to fetch and decode node stats" err="failed to get cluster health from https://elastic:9200//_nodes/_local/stats: Get https://<prom-user>:<prom-password>@elastic:9200/_nodes/_local/stats: remote error: tls: handshake failure" <time> elastic elasticsearch_exporter: level=warn ts=<time> caller=nodes.go:1027 msg="failed to fetch and decode node stats" err="failed to get cluster health from https://elastic:9200//_nodes/_local/stats: Get https://<prom-user>:<prom-password>@elastic:9200/_nodes/_local/stats: dial tcp <ip>:9200: getsockopt: connection refused"

zwopir commented 6 years ago

Hi Ben,

I don't have a corresponding setup in place, so I can't reproduce the issue. Can you make sure that the setup is correct? Can you curl the endpoints mentioned in the exporter log? Connection refused sounds more like a configuration mismatch than a bug to me, but I might be wrong

benno001 commented 6 years ago

Hi Christoph,

I tested curl - that seems to work correctly. The important part is probably the 'tls: handshake failure': after re-deploying my test environment, that's the only error I see on the exporter side.

This is in the elastic logs: [2018-07-12T12:58:14,048][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [elastic.local] caught exception while handling client http traffic, closing connection [id: 0x5fd2954b, L:0.0.0.0/0.0.0.0:9200 ! R:/<ip>:40872] io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: no cipher suites in common at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?] at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?] at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_171] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:255) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1162) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?] ... 15 more Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?] at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1115) ~[?:?] at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:807) ~[?:?] at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:228) ~[?:?] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?] at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?] at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_171] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?] at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1301) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1214) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?] ... 15 more

zwopir commented 6 years ago

Hi Ben,

I'm currently on vacation with my kids and unfortunately can't really dig into the issue. You can find the supported cipher suites here: https://golang.org/pkg/crypto/tls/

Maybe we can find the issue there...

zwopir commented 6 years ago

Hi Ben, did my link to the supported ciphers help solving/identifying the issue?

benno001 commented 6 years ago

Hi Christoph, Unfortunately I didn't get it to work, but it seems the ciphers should be supported on both ends. Possibly this is a mismatch in implementation between Java/Go..

2018-08-06 10:49 GMT+02:00 Christoph Oelmueller notifications@github.com:

Hi Ben, did my link to the supported ciphers help solving/identifying the issue?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/justwatchcom/elasticsearch_exporter/issues/158#issuecomment-410635562, or mute the thread https://github.com/notifications/unsubscribe-auth/AFvze4K5EtKDJSj6hNFvXNrcqjrpSFHsks5uOAMxgaJpZM4VMiUE .

zwopir commented 6 years ago

I'm sorry I can't help. I keeping this issue open, but currently don't have the time to further investigate it. As a workaround I'd propose to put a nginx in between the exporter and prometheus