Howto: Elasticsearch with custom SSL setup

[codefromthecrypt]

Marking this as a question, for now. We have someone on gitter stanltam_twitter, asking about how to use custom TLS configuration for the client side of elasticsearch.

I know this exists for the server side, but I'm not sure if we read custom TLS or not, if we fall back to system properties, or something else.

https://github.com/line/armeria/blob/master/spring/boot-autoconfigure/src/main/java/com/linecorp/armeria/spring/ArmeriaSettings.java#L35-L49

Ex I would expect possibly some custom TLS object used here https://github.com/openzipkin/zipkin/blob/master/zipkin-server/src/main/java/zipkin2/server/internal/elasticsearch/ZipkinElasticsearchStorageConfiguration.java#L75

Holding back a patch release for now, just in case we have to change code to make it possible to connect to ES with customized TLS

cc @trustin @anuraaga

[anuraaga]

Yeah I think we'd need to add settings for custom TLS for the client. I don't think we can really share settings between the server and client since they're usually configured differently and with different settings but we should add some settings to ElasticsearchProperties.

It's a little tempting to reuse this object, but it has server-only setting of ClientAuth so probably shouldn't be used for configuring a client https://github.com/line/armeria/blob/master/spring/boot-autoconfigure/src/main/java/com/linecorp/armeria/spring/Ssl.java

[codefromthecrypt]

ok so it makes sense then probably to just share the property keys and not the server-only one.

do you feel confident and/or have cycles for this? (ideally both :)) either way is ok

On Fri, Aug 23, 2019 at 7:41 PM Anuraag Agrawal notifications@github.com wrote:

Yeah I think we'd need to add settings for custom TLS for the client. I don't think we can really share settings between the server and client since they're usually configured differently and with different settings but we should add some settings to ElasticsearchProperties.

It's a little tempting to reuse this object, but it has server-only setting of ClientAuth so probably shouldn't be used for configuring a client

https://github.com/line/armeria/blob/master/spring/boot-autoconfigure/src/main/java/com/linecorp/armeria/spring/Ssl.java

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin/issues/2774?email_source=notifications&email_token=AAAPVV3QQOVE3RP54RFRVGDQF7EHTA5CNFSM4IO6VYCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD477BDI#issuecomment-524284045, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAPVV3RXAI4SMDDSGRTX23QF7EHTANCNFSM4IO6VYCA .

[anuraaga]

Will say probably no cycles. It seems like a new feature to configure TLS of the ES client, rather than bugfix, and ends up below other things in my personal backlog right now. I wonder a little if it also blocks the patch release for the same reason.

[anuraaga]

Actually realized we might already be able to support the javax.net properties if the user disables openssl using the flag (sorry walking/ texting so haven't looked up the link to the flag yet)

[codefromthecrypt]

@anuraaga good ideas.. mainly I think the old one used normal javax.net so in comparison not being able to, and having no other way to, could look like a regression. we can release a patch, but still I wouldn't announce 2.16 until it is stable with things like this.

(doesn't mean you have to do this.. I can add the ssl parameters it seems not difficult to copy/paste and I'm at the computer)

[codefromthecrypt]

I'm going for it :P

[codefromthecrypt]

from chat the only settings used were: keyStore, keyStorePassword, trustStore, trustStorePassword

So, it was more about overrides than client auth.

[codefromthecrypt]

ps I'm getting some 🍷 then coming back to this. approach is to add ssl into the ES config properties object, defaulting to the system properties. That way it is 2-for-1. Then wire into the ssl configurator for the client. I've already done all this..

the wine is needed to refactor the integration test :P luckily I have an old keystore and some things from my past life in feign. I'll cobble it together in a bit.

[codefromthecrypt]

https://github.com/openzipkin/zipkin/pull/2775 should work.. waiting for manual verification as I don't feel like setting up elasticsearch ssl

[laijianbin]

@anuraaga can you help check this problem After I execute JAVA_OPTS='-Djavax.net.ssl.keyStore=elastic-certificates.p12 -Djavax.net.ssl.keyStorePassword=123456 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.trustStore=elastic-certificates.p12 -Djavax.net.ssl.trustStorePassword=123456 -Djavax.net.ssl.trustStoreType=PKCS12' KAFKA_BOOTSTRAP_SERVERS=172.17.2.154:9092 KAFKA_TOPIC=bos-dev-zipkin STORAGE_TYPE=elasticsearch ES_HOSTS=https://172.17.2.154:9200 ES_USERNAME=elastic ES_PASSWORD=123456 EX_INDEX=bos-dev-zipkin java $JAVA_OPTS -jar zipkin-server-2.19.1-exec.jar and visit http://localhost:9411/zipkin/api/v2/services

javax.net.ssl.SSLHandshakeException: error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1002) ~[netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1268) ~[netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1222) ~[netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1293) ~[netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1336) ~[netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:204) ~[netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) [netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1239) [netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1276) [netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) [netty-codec-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) [netty-codec-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) [netty-codec-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:154) [netty-handler-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502) [netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) [netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar!/:4.1.43.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.43.Final.jar!/:4.1.43.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

[anuraaga]

The command looks fine - there are any number of reasons that could cause handshake failures such as expired or incorrect certificates, so it's always hard to pinpoint one. Does this only happen in a particular version of zipkin or even in older ones? Also, are you using Java 11? If so, can you try with the system property -Dcom.linecorp.armeria.useOpenSsl=false and see if anything changes?