plugins https/ssl options inconsistencies

[colinsurprenant]

We should probably look into using a standard config option for using the https/ssl protocol. I did a little survey and this is what we currently use:

• *_protocol => ['http', 'https’] - logstash-output-elasticsearch-ec2
• ssl => boolean - logstash-output-elasticsearch
• ssl => boolean - logstash-input-rabbitmq
• proto => ['http', 'https’] - logstash-output-loggly
• ssl_enable = > boolean - logstash-input-tcp
• ssl => boolean - logstash-output-redmine
• enable_ssl => boolean - logstash-output-riak

I suggest we standardize on ssl:

config :ssl, :validate => :boolean

[jordansissel]

I wonder if we shouldn't rename it to tls ? (since SSL <= 3.0 is deprecated)

[jordansissel]

Or for technologies not using TLS, can we call it secure more generally?

[colinsurprenant]

good point.

[colinsurprenant]

I think I like secure since, as you say, it can also include non TLS/SSL use cases...

[jordansissel]

Thinking out loud -

Since secure is the intent, but the specific meaning will vary from plugin to plugin, it may be worth requiring each plugin's secure setting to document exactly what technology is used and what functionality is provided (trust, authentication, encryption, etc). This seems doable given we'll have to go through each plugin to add this setting anyway.

I'd also like to see, long term, unified security settings for credentials (user/pass, certificates) and trusts (IP lists, certs, etc), but that's outside the scope of this ticket.

[jsvd]

If ssl becomes the norm, then we need to change:

*_protocol => ['http', 'https’] - logstash-output-elasticsearch-ec2 proto => ['http', 'https’] - logstash-output-loggly ssl_enable = > boolean - logstash-input-tcp enable_ssl => boolean - logstash-output-riak

[jordansissel]

+1 on secure w/ documenting exactly what security technology is used and functionality provided.

[purbon]

like the @jordansissel option, but we should have a clear note on the tech used for that, cause secure is so vague.

[suyograo]

I like secure too.

[jordansissel]

Here's my full proposal for unification of the secure boolean setting. All of the following:

• use secure for enabling any security-related features if security is something toggleable for an input/output
• add backwards-compatibility to have ssl => true (or whatever the current setting is) to imply secure => true AND mark the old setting :deprecated => "Use 'secure => true'"
• document precisely which technology and function is enabled when secure is true
• if additional input is required (credentials, etc), further validation must be added if secure is true.
• add tests to verify that missing credentials/etc will cause config failures if secure => true

[colinsurprenant]

:+1:

[purbon]

:+1:

[runningman84]

This is linked to this issue: https://github.com/elasticsearch/logstash/issues/2394

[colinsurprenant]

new major version 3.0 is a good opportunity to refactor option name.

[purbon]

+1 on this, having a uniform syntax and experience would be awesome. Might be having a common mixin?

On Wed, 13 Jan 2016 23:26 Colin Surprenant notifications@github.com wrote:

new major version 3.0 is a good opportunity to refactor option name.

— Reply to this email directly or view it on GitHub https://github.com/elastic/logstash/issues/2151#issuecomment-171456972.

[harbulot]

@ph, thank you for pointing out this issue. I'm not sure whether it would be relevant to copy again the background information I've already written for logstash-plugins/logstash-input-tcp#47.

I've also noticed that some plugins (at least logstash-input-http) seem to use a more traditional JSSE approach: keystores and truststores.

Taking a step back, I'm just wondering what the benefits of using an OpenSSL-like approach (via JRuby-OpenSSL) are. Is it to keep some compatibility with the MRI in the long term, or is it just for historical reasons because many libraries used in Logstash were effectively Ruby libraries using Ruby's OpenSSL module? If it's the latter, would a switch to a JSSE approach be worth considering or even possible at all? (I must admit I don't know much about Ruby or JRuby, so I don't know if it would make sense at all.) There could still be ways of using certificates in PEM format using a more JSSE-based approach, with a small additional layer of code.

---

@jordansissel, one of the problems with "secure" is that it's rather vague. There can be various security levels, which will vary depending on the environment where it is deployed: what's "secure" for someone might not be secure for someone else. It could either give the false impression that something is secure when it is not (e.g. some people still treat "using SSL" as being equivalent to "secure", whether or not they're verifying the server certificate, hostname, or using adequate cipher suites), or activate certain settings that might not be necessary in the user's environment (e.g. using client-certificate authentication), in which case you could have something with "secure => true" and also another option to turn off certain settings configured at the same time, which would make the overall configuration quite hard to understand. Having sensible default values for the configuration options is probably a better way.

[jordansissel]

one of the problems with "secure" is that it's rather vague

I agree, though if we discuss terminology, then "SSL" is an incorrect term anyway, since we shouldn't even support SSL v3 or older. I would like to see if we can use reasonable terminology that encourages users to do secure things. For example, many many many folks enable TLS but disable certificate verification (VERIFY_NONE, etc) simply because that makes data flow and then they can stop worrying about it (despite it being insecure). I don't know what the right naming is right now, but I am certain that SSL/TLS nomenclature is basically garbage to most folks and the whole space is very difficult to understand and configure correctly. Let's see if we can make it approachable.

[harbulot]

Yes, I agree. I think the "SSL" broad term is here to stay whether we like it or not: firstly, pretty much all SSL/TLS libraries or tools with SSL/TLS settings use "SSL" to refer to classes, functions or settings that have something to do with the SSL/TLS stack in the broad sense; secondly, some tools use "TLS" to refer to the mechanism of upgrading from plain to SSL/TLS over an existing socket via a STARTTLS-like mechanism.

I think we can achieve something approachable by using sensible default values, activated when ssl_enable is true:

• Sensible default protocols and list of cipher suites.
• On the client:
  • Verifying the server certificate, at least with the default system CA certificates.
  • Verifying that the server identity matches what we were expecting (hostname verification: one of RFC 2818, RFC 6125 or RFC 5425. The RFC 2818 options can be a sensible default if no implementation of RFC 6125 is available).
• On the server:
  • I wouldn't necessarily enable client-certificate authentication by default: it generally requires a fairly good understanding of PKI to be used, and if the whole system CA certificates are to be trusted, it can create a substantial overhead in the handshake.
  • For many plugins like logstash-input-tcp, client-certificate authentication would certainly be used as a form of authorisation directly. Hence, mandating a client certificate (instead of making it optional) would often make sense in the Logstash context. That is, ssl_verify => true on the server should imply that the certificate is requested, but also subsequently required, which is already the case, but there could also be an ssl_verify => optional which would request a client cert but accept the connection if it's not presented. (I'm mentioning this because I think I've seen some SSL/TLS issues on some of the plugins about having the ability to make client-certificate verification optional.)

What would remain would be options to configure the cert chain + key (or keystore), but there can't be any default value for that anyway (unless a self-signed cert is automatically generated, but that more or less implies the remote party would have to trust that certificate blindly by default, which isn't ideal).

Other options that could make sense would be to check for revocation. Since this relies on the JSSE, CRL distribution point and OCSP should be available out of the box (assuming a PKIX trust manager is used, I'm not sure what JRuby-OpenSSL uses) when setting the right system properties. That said, the whole system is far from ideal, so turning in on by default might not be the best option. My guess is that many Logstash deployments will use an internal/corporate CA anyway, so it might be worth adding options to deploy CRLs locally (similarly to SSLCARevocationFile in Apache Httpd).

[ph]

Taking a step back, I'm just wondering what the benefits of using an OpenSSL-like approach (via JRuby-OpenSSL) are. Is it to keep some compatibility with the MRI in the long term

This is my mid term goal, @jordansissel and myself had some discussions how we can improve the situation with SSL/Jruby, I think as we move forward more thing will be written in java to leverage the java standard libraries. The innovations and experiments will probably come from the beats inputs, since this is were I am playing with them and this is also were people want more security.

Like you have seen the Jruby-OpenSSL gives us less flexibility in what we can achieve.

Verifying the server certificate, at least with the default system CA certificates.

I think we shouldn't do that and require it to be more explicit, in correctly configured system the admin will make sure the trusted CA of the system are the right one. But in the common world I think its safer to ask to a specific files or path.

[ph]

Concerning the naming, I think beats had done it right with their options. https://www.elastic.co/guide/en/beats/filebeat/current/configuration-output-tls.html

They are basically exposing everything that the golang TLS implementation can offer, but they still provide sane default when nothing is specific and insecure is not set to true. insecure is just the verify_mode set to none.

[ph]

I wouldn't necessarily enable client-certificate authentication by default: it generally requires a fairly good understanding of PKI to be used, and if the whole system CA certificates are to be trusted, it can create a substantial overhead in the handshake.

Agree this is somewhat an advance feature and in most situation the verify the server is enough.

[sgpinkus]

Concerning the naming, I think beats had done it right with their options.

Agree. The way beats configure SSL to logstash, or to elasticsearch, or kibana to elasticsearch all the same. Ex https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html#configuring-ssl-logstash. Logstash for some reason still only accepts a "keystore" (AFAIK reading the config doc). Very hard for people outside Java developers to understand how that maps to a client SSL key and cert.

[jsvd]

Another item to consider here is the behaviour of compression for plugins that support compression and TLS (e.g. HTTP input and Elasticsearch plugins, kafka). Enabling compression by default raises some concerns wrt the BREACH attacks on TLS when HTTP compression is being used. That said, there is a set of conditions for the attack to be applicable (more here https://www.breachattack.com/).

We can either decide to make all plugins behave the same, or have different defaults depending on the plugin and use case. For a reference default behaviour we can use what Elasticsearch does: enable compression by default unless TLS is used.