[discuss][Fleet] Custom CAs and SSL options

ph commented 4 years ago

Summary of the problem

Having secure communication between Elastic-Agent and elasticsearch or tier services requires to deal with SSL (TLS), which is complicated, have a lot options and easy to get wrong. We would need to be able to configure the following options on outputs and integrations that support SSL.

Certificate authorities (common): when authenticating the server.
Certificate: required when doing mutual authentication.
Certificate private key: required when doing mutual authentication.
key_passphrase: used to decode the private key.
verification_mode (advanced): Control how the client verifies the server.
supported protocol (advanced): define which version of TLS to use.
cipher_suites (advanced): Define which cipher is supported, required in some lock down environment.
curve_types (advanced): Define which ECDH curve to use.
renegotiation (advanced): Configure TLS renegotiation
client_authentication (advanced): Configure client authentication.
ca_sha256 (advanced): This configures a certificate pin that you can use to ensure that a specific certificate is part of the verified chain.

For a complete list of types see Filebeat documentation.

The Certificate authorities, Certificate and certificate private key are more often files or blob of text. It's also common to be able to reuse the artifacts on multiples integrations or outputs, and it's convenient to only have to update a single file on disk.

User stories:

As a Fleet administrator I should be able to define the Certificate Authorities, Certificate and the private key on outputs.
As a fleet administrator I should be able to define the Certificate Authorities, Certificate and the private key on integrations that require it.
As fleet administrator I should be able to define SSL options.

Other

Maybe we could prepopulate options for known service like Elasticsearch because its already configured in Kibana.

elasticmachine commented 4 years ago

Pinging @elastic/ingest-management (Team:Ingest Management)

ph commented 4 years ago

FYI @nchaulet

ph commented 4 years ago

@hbharding TLS/SSL options are not clear, ping me if you need clarification.

ph commented 4 years ago

@mostlyjason @ruflin I would like your inputs on this, should we try to make a "Secret store" and allow users to reference values from it or should we expect user to fill out that information and consider that information not to be reusable? Maybe we can later first and add reusable later when we deal with reusable configuration?

ruflin commented 4 years ago

Lets start with not reusable / per config first and make it reusable later on. I don't think we will get around it.

To not have to start with a complex form, perhaps we could first allow users to configure it in free form / yaml?

mostlyjason commented 4 years ago

Sorry for the delay +1 on what @ruflin said. I think the user persona for this feature is advanced-level platform owners, and we can expect them to be very technical and familiar with yml config since its required for elasticsearch and kibana. I assume basic users can use cloud where this is set up automatically.

nicpenning commented 4 years ago

Hello,

An advanced-level platform user looking to use the new agent!

Anything we can do to help escalate this functionality? Currently Elastic is knocking down any Elastic agent requests because of the self signed certificate.

ruflin commented 4 years ago

@nicpenning Great to hear you are trying it out. Which of the above proposed config options would you need for your use case?

ruflin commented 4 years ago

@nicpenning It seems @jamiesmith found a workaround for now, does this also work for you? https://github.com/elastic/kibana/issues/73483#issuecomment-676419501 This does not mean, we shouldn't fully support it in the future BTW.

nicpenning commented 4 years ago

Right now we deploy winlogbeat at scale by unzipping a file that has the exe, yml, cert, key, etc.. and we configure the yml to use the CA, Cert, and Key so all three are required for mutual TLS authentication and secure transport.

The certs shouldn't change that often whether they are self signed or created with our organization CA. But they could expire over time so it could be a ticking time bomb if the certs expire and need to be updated for the agents to connect. I am not sure how to handle that part.

For deployment purposes I think I would lean towards: As a Fleet administrator I should be able to define the Certificate Authorities, Certificate and the private key on outputs.

Let's say I could define the CA, cert and key today, it sounds like I would have to get the elastic agent zipped download and include the CA, cert, and key so I could then "deploy" the agent with all the necessary files. This would be an extra step in the process but would ensure that nothing could send logs unless they had the additional files.

nicpenning commented 4 years ago

@ruflin I might be a little confused because we had no issues enrolling the agent. The --insecure or giving the Kibana CA allowed us to enroll. But on the Elastic ingest side, it seems that Elastic won't trust the agent because the agent does not have the ca, cert, and key pairs to allow that to happen. But I could misguided if it doesn't work that way.

I see: [2020-08-20T00:00:01,341][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.4.79:9200, remoteAddress=/192.168.205.7:53370} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:167) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:729) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:684) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:499) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:475) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] ... 16 more [2020-08-20T00:00:07,032][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.4.79:9200, remoteAddress=/192.168.205.7:53374} io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16) at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:325) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:268) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:132) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:729) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:684) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:499) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:475) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] ... 16 more Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16) at sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1895) ~[?:?] at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240) ~[?:?] at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197) ~[?:?] at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:729) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:684) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:499) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:475) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] ... 16 more

nicpenning commented 4 years ago

The Elastic node settings that pertain to security look like this:

xpack.security.enabled: true xpack.security.authc.api_key.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: 'certs\elastic1.p12' xpack.security.transport.ssl.truststore.path: 'certs\elastic1.p12' xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: 'certs\elastic1.p12' xpack.security.http.ssl.truststore.path: 'certs\elastic1.p12' xpack.security.http.ssl.verification_mode: none

Note, I tried to set the SSL verification mode to None but did not seem to make a difference. We also have a a password on the .p12 and it is stored in the secure key store.

nicpenning commented 4 years ago

@ruflin That work around worked for me! Thank you!

ruflin commented 4 years ago

@nicpenning Great to hear you found a workaround!

jen-huang commented 3 years ago

FYI Synthetics opened #95220 for us to investigate a common SSL component for integrations, I closed that one to roll it in here. @mostlyjason This hasn't been on our radar in the last few releases or the upcoming one, do you think this is still needed in 7.x?

mostlyjason commented 3 years ago

We need to document the output settings https://github.com/elastic/observability-docs/issues/586. This should unblock users. A shared component for settings is an interesting idea but seems like an enhancement. We still have other blockers on our roadmap we need to address, so I'd come back to this after 7.x.

elastic / kibana

[discuss][Fleet] Custom CAs and SSL options #72718