Simplifying ssl/tls configuration (server)

[cescoffier]

Enabling SSL (TLS actually) on the server and client is not particularly easy. While I was working on SNI support, I also found different approaches in various extensions.

Current approach

The Vert.x HTTP extension handles the configuration of TLS. The SSL configuration is stored under quarkus.http.ssl. Note that the SSL port is under quarkus.http.sslPort. SSL is enabled as soon as the SSL config contains certificates.

First, we are still referring to SSL. However, it is not the correct wording, as we are using TLS.

To use SSL / TLS, we need certificates. These are configured, in the Vert.x HTTP extension, under: quarkus.http.ssl.certificates (gRPC does not have this certificates sub-level). There are multiple formats: PEM, JKS, and PCKS12, all with different configurations. The current configuration is a direct mapping of the Vert.x configuration, but it might be confusing. Typically, to use a PEM certificate and its key, you configure:

quarkus.http.ssl.certificates.file=path-to-cert-file.pem
quarkus.http.ssl.certificates.key=path-to-key-file.pem

For JKS, you use:

quarkus.http.ssl.certificate.key-store-file=path-to-server-keystore.jks
quarkus.http.ssl.certificate.key-store-password=secret

PKCS12 uses the same attribute as JKS:

quarkus.http.ssl.certificate.key-store-file=path-to-server-keystore.pkcs12
quarkus.http.ssl.certificate.key-store-password=secret

The key store password is optional, but use the rather odd default: "password".

All the paths refer to files that should not be packaged within the application (for security reasons).

When key-store-file is used, you can configure the type of key store. If not set, Quarkus checks the extensions of the file. Note that it does not cover PEM.

You also need to configure the trust store:

quarkus.http.ssl.certificate.trust-store-file=server-truststore.jks
quarkus.http.ssl.certificate.trust-store-password=password

Only JKS and PCKS12 are supported for the trust store. If the type is not set, it makes a guess based on the file name. For the trust store, there is no default password.

While not yet supported (PR pending), SNI changes the configuration a bit, as in this case, you can have multiple PEM files (keys and certificates). Other formats support multiple certificates.

Proposal

This issue is about discussing a new simpler format.

quarkus.http.tls.key-store=path to jks/pcks file
quarkus.http.tls.key-password= ... no default, mandatory
quarkus.http.tls.key-store-type= optional, file based guess

Same for the trust store:

quarkus.http.tls.trust-store=path to jks/pcks file
quarkus.http.tls.trust-password= ... no default, mandatory
quarkus.http.tls.trust-store-type= optional, file based guess

What about pem?

As the pem format is slightly different and can contain multiple entries (SNI), I would rather go for a map:

quarkus.http.tls.pem.1.file=path to pem file
quarkus.http.tls.pem.1.key=path to key pem file
quarkus.http.tls.pem.2.file=path to the second pem file
quarkus.http.tls.pem.2.key=path to the second key pem file

Final considerations

Note that the proposed configuration switch from SSL to TLS, as SSL 2.0 is deprecated since 2011 and SSL 3.0 since 2015. Also, that configuration does not use a certificates nested config object.

Another aspect to discuss is how we can avoid breaking configuration. The fact that the proposal uses a different config group (TLS) can allow us to deprecatessl.* and recommend switching to tls.*.

The idea would be to apply the same structure to all our extensions supporting TLS (gRPC for example).

Thoughts?

\CC @geoand @stuartwdouglas @gsmet @radcortez @ebullient

[geoand]

Absolutely +1 from aligning and for deprecating the old stuff we have

[radcortez]

Agree.

Another aspect to discuss is how we can avoid breaking configuration. The fact that the proposal uses a different config group (TLS) can allow us to deprecatessl.* and recommend switching to tls.*.

In SR Config we can relocate or fallback property names, so you can link the old and the new name. Relocate will use the new property value if you refer to the old value. Fallback will use the old value if you can't find the new one.

[sberyozkin]

@cescoffier Hi Clement - Did you mean using quarkus.http.ssl.certificate.key-store-type instead of quarkus.http.ssl.certificate.key-store-file-type ? This is fine, just FYI, the type and also a provider are configured for the BC FIPS JSSE test:

quarkus.http.ssl.certificate.key-store-file-type=BCFKS
quarkus.http.ssl.certificate.key-store-provider=BCFIPS

[cescoffier]

@sberyozkin good point about FIPS. I would drop the "file-type" and just use "type. provider will still be supported of course (but rather an advanced used case, I don't know any other usage than FIPS).

So for FIPS, it would be:

quarkus.http.ssl.key-store-type=BCFKS
quarkus.http.ssl.key-store-provider=BCFIPS

[ebullient]

plus a million for simplifying and moving to "modern" terminology. The new config space means we can do whatever coping is required if the old properties are still present (I don't think it is as simple as just mapping to new values).

Do we need a separate configuration for http vs. gRPC?

should we drop to quarkus.tls. ... and have one thing that handles cert stuff? I am thinking off the cuff w/o enough coffee, so that may be silly.

[cescoffier]

Do we need a separate configuration for http vs. gPRC? should we drop to quarkus.tls. ... and have one thing that handles cert stuff? I am thinking off the cuff w/o enough coffee, so that may be silly.

Good question. It's two different servers (two different ports). Also, the Vert.x HTTP server has more features than the gRPC one (typically, the gRPC server does not support SNI). So, I would lean toward having 2 different configurations BUT using the same config structure (same key name, same management behind the hood).

[stuartwdouglas]

@ebullient I also like the idea of having an (internal) TLS extension that handles all the config, and other susbystems just consume the results. It means TLS config could be unified for all extensions.

[cescoffier]

It would need to distinguish the server and client. So, we could have quarkus.server.tls, but quarkus.tls could be problematic.

I'm not a big fan of 'server' as we may have servers not reading this configuration. But, can't find anything better.

Also, there is one tricky case. Imagine http and gRPC, both configured with this common configuration. If HTTP, supporting sni uses multiple pem files, gRPC would only consider the first one (at least for now). It might be fine, and a simple warning enough, but that's something we should consider.

[ebullient]

Maybe TLS config is a map (not single) with named profiles, so that consumers could do: quarkus.grpc.tls.profile="noodles" (I have had no coffee yet, no comment on ludicrous examples).

I would use quarkus.tls.*. I think server vs. client certificate usage could work, or we could just stick with profiles, and then other config references those for usage.

Also means we would have a place to set up letsencrypt support and deal w/ cert rotation and other things without having to reproduce that config in different config namespaces.

[cescoffier]

Also means we would have a place to set up letsencrypt support and deal w/ cert rotation and other things without having to reproduce that config in different config namespaces.

+1

I'm a bit concerned about introducing profiles here (confusion with the quarkus profile). But at the same time, named configuration could we great.

[ebullient]

yes. agree re: confusion with profile word. Naming things sucks. As I said, it was early. ;)

[cescoffier]

So basically, there are two proposals. I will try to summarize them in this comment and add pros/cons for each. Both aim to simplify and provide a cohesive experience when dealing with TLS.

Proposal A - Each extension owns the configuration

This approach is mostly a cleanup of the current situation. Each extension would continue to handle SSL/TLS. The only difference is that extensions would share a common Config object and have access to utility classes to load the certificates. Basically, for the HTTP server it would provide something like:

quarkus.http.tls.key-store=path to jks/pcks file
quarkus.http.tls.key-password= ... no default, mandatory
quarkus.http.tls.key-store-type= optional, file based guess
quarkus.http.tls.trust-store=path to jks/pcks file
quarkus.http.tls.trust-password= ... no default, mandatory
quarkus.http.tls.trust-store-type= optional, file based guess

# PEM file
quarkus.http.tls.pem.1.file=path to pem file
quarkus.http.tls.pem.1.key=path to key pem file
quarkus.http.tls.pem.2.file=path to the second pem file
quarkus.http.tls.pem.2.key=path to the second key pem file

In addition to these fields, the shared configuration would allow configuring: the cipher suites - if not set, use a sensible default protocols - if not specified, use a sensible default trust store provider and file type key store provider and file type SNI (enable or disable it) The shared utilities would allow reading the certificates and other files.

Note that each extension would be responsible for validating the configuration.

This approach is in the same vein as the current approach but rationalizes the format.

Proposal B - TLS extension

This second approach is slightly different as it introduces an extension managing TLS on behalf of the other extension. These other extensions (HTTP, gRPC) would depend on it to access the configured certs.

That extension would be configured using named buckets (as I can't find a better name, I picked the worst one). So, if we consider the http and grpc bucket, we would get:

quarkus.tls.server.http.key-store=path to jks/pcks file
quarkus.tls.server.http.key-password= ... 
quarkus.tls.server.http.trust-store=path to jks/pcks file
quarkus.tls.server.http.trust-password= ... 

quarkus.tls.server.grpc.key-store=path to jks/pcks file
quarkus.tls.server.grpc.key-password= ... 
quarkus.tls.server.grpc.trust-store=path to jks/pcks file
quarkus.tls.server.grpc.trust-password= ... 

The extension handles a Map<String, TlsServerConfig>, with the TlsServerConfig being close to the first proposal. Other extensions would use this extension to access the TlsConfig, using either a default name or a specific name:

quarkus.http.tls=my-tls-bucket-name

The same could be applied to the clients. However, the client configuration is slightly different, so that it would need a separate Map (that's why the proposal keys have .server.).

While this proposal is more disruptive, it also has more perspectives. Typically, we can start thinking about a proper way to handle certificates and certificate generation and renewal (Acme). On the other side, the configuration might be more convoluted. Each extension will still need to validate the structure if it does not support some feature (like an extension not supporting pem files). It also reduces the ability to use custom parameters for each extension.

CC @ebullient , @geoand , @stuartwdouglas

[geoand]

Thanks for the great summary @cescoffier.

Personally I am leaning towards the first proposal.

[stuartwdouglas]

Personally I like the idea of the second proposal. My thinking is that this could make things a lot easier to configure if you have your own CA. If you can just specify your truststore once and then have everything use it this could simplify things, although how this would work in practice I am not sure.

I did get a query about postgres + SSL the other day, and the driver only supports adding a path to a certificate in the URL. If we have fully unified config we could potentially automatically make this work as well, so there really is only one place to configure SSL.

@darranl bassically did approach 2 for Elytron in WildFly so he might also have some insight.

[geoand]

Personally I like the idea of the second proposal. My thinking is that this could make things a lot easier to configure if you have your own CA. If you can just specify your truststore once and then have everything use it this could simplify things, although how this would work in practice I am not sure.

I did get a query about postgres + SSL the other day, and the driver only supports adding a path to a certificate in the URL. If we have fully unified config we could potentially automatically make this work as well, so there really is only one place to configure SSL.

But do you really want to configure SSL in place in all cases?

@darranl bassically did approach 2 for Elytron in WildFly so he might also have some insight.

I'd really like to hear Darran's insight on this

[radcortez]

I'm also in favour of Proposal B, but with some reservations as well.

On the other side, the configuration might be more convoluted. Each extension will still need to validate the structure if it does not support some feature (like an extension not supporting pem files).

This indeed may hurt the user experience. Lets imagine two extensions, one supports the pem file and the other one does not. We cannot really fail the build, we can add a warning for the extension that does not support the pem file, but the user may not notice it and expect it to work.

It also reduces the ability to use custom parameters for each extension.

We should be able to add custom parameters for each extension. Something like:

quarkus.tls.server.http.extension.property.a
quarkus.tls.server.http.anotherextension.property.b

[stuartwdouglas]

If possible we should try and add support for things that extensions don't support natively, e.g. write a TrustStore that supports PEM files and provide that to extensions that don't know how to deal with PEM files.

[ebullient]

Exactly, Stuart. One place that can deal w/ the nuances of TLS/SSL would be epic. Quarkus extensions are all about dealing with imposed constraints and mismatches (for native mode or anything else), and if that makes for better / more consistent TLS configuration and behavior (especially if it makes it easier to integrate w/ cloud native environments, secrets, vaults, .. ), that is a solid win for users.

[stuartwdouglas]

Yea, I think a TLS extension will likely be more work for us, but the end result will be a much better experience for users.

[cescoffier]

The important part is the ability to add features to this new extension such as ACME, revocation, and so on. Adding that Today would require changing multiple extensions. So, yes, more work, but somewhat better in the long term.

[geoand]

If we do have a single TLS extension, what would configuration look like if a user needs to configure a certificate for listening on HTTPS and also a trust store for the rest client (to call a downstream service)?

[cescoffier]

I guess it would use the "named" bucket:

# HTTPS
quarkus.tls.server.http.key-store=path to jks/pcks file
quarkus.tls.server.http.key-password= ... 
quarkus.tls.server.http.trust-store=path to jks/pcks file
quarkus.tls.server.http.trust-password= ... 

quarkus.tls.client.my-rest-service.trust-store=path to jks/pcks file
quarkus.tls.client.my-rest-service.http.trust-password= ... 

my-rest-service would match the configKey of the rest client.

[stuartwdouglas]

IMHO it should be like databases, where there is a default unnamed one (one for client, one for server), and then you can specify named ones if you need more flexibility.

[geoand]

I must be missing something, but if we do use a default one, how do you determine where it applies?

[cescoffier]

For servers, we can either use a default one + custom ones

quarkus.tls.http.key-store=path to jks/pcks file
quarkus.tls.http.key-password= ... 
quarkus.tls.http.trust-store=path to jks/pcks file
quarkus.tls.http.trust-password= ... 

quarkus.tls.my-custom-cert.http.key-store=path to jks/pcks file
quarkus.tls.my-custom-cert.http.key-password= ... 
quarkus.tls.my-custom-cert.http.trust-store=path to jks/pcks file
quarkus.tls.my-custom-cert.http.trust-password= ... 

Or we define a default name for all extensions (http, grpc`...).

The first approach would allow reusing configuration between servers (which would also be possible using names).

[stuartwdouglas]

It would apply everywhere that has enabled SSL and has not explicitly specified a named context.

[geoand]

It would apply everywhere that has enabled SSL and has not explicitly specified a named context.

Ah, OK. I was missing the enabled part and thinking it would apply everywhere...

[cescoffier]

Ok, so we probably should create an epic out of this discussion.

[cescoffier]

FYI, I'm going to try to start working on this...

[maxandersen]

related to the fun around https://github.com/quarkusio/quarkus/issues/8975 for having just one place to skip verification but allow turning it on in specific places.

That is nice!

One question I have about use then having keystores separately from javas main pem store. We would still be able to pass in at runtime a setting/property to get the certificates for the cluster managed by something like https://cert-manager.io/docs/release-notes/release-notes-1.2/#usability-improvements ?

That cert you would want all the subsystems to be aware of if I grok it right ...but you write as if client and server would need to know different certs ? how will that add up?

[cescoffier]

cert-manager is able to provision a set of certificates and you would have to "map" them to your different client/server. A bit like SBO.

[KarlScheibelhofer]

If you need a KeyStore implementation for PEM format, you can have a look at https://github.com/KarlScheibelhofer/java-crypto-tools . I have written this recently and would contribute it. Currently, it supports reading PEM files with private keys and certificates and basic creation.

[KarlScheibelhofer]

Here https://github.com/KarlScheibelhofer/jcp-demo-quarkus-https you can find a simple demo showing how to use java-crypto-tools to configure a PEM format keystore in quarkus.

In addition to using a PKCS12 keystore, you can see these differences:

• key-store-file-type in application.properties is PEM rather than PKCS12.
• Additional dependency in pom.xml to dev.scheibelhofer:crypto-tools
• A @QuarkusMain class to install the JctProvider.
• key-store-provider in application.properties is JctProvider

If Quarkus integrates this lib, only setting the key-store-file-type to PEM would be left.

[cescoffier]

For the record, I restarted working on this more seriously recently.

[maxandersen]

39825 last comment states there is more to do - should this stay open yet or ?

[cescoffier]

I think we can keep it closed - the mechanism is in place and it's now just a question of adding support into extensions. I have already fixed the mailer. Redis and Rest client are next.

Let me create separate issue for the missing ones (I have the list - it's not long).