Add support for SSL certificate generation

[mp911de]

Add support to request SSL certificates from Vault's PKI backend. Certificates can be issued on demand by using a role.

The generated certificate and the issuing CA certificate need to be stored in a Truststore (Keystore) and the private key in a Keystore. Vault should be used as intermediate CA, so the Truststore should contain the Root CA certificate that should be configured along the certificate request properties.

Config parameters for requesting certificates:

• common-name
• alt-names
• ip-sans

Other required parameters:

• Root certificate

Challenges:

• Public key pinning: Preserve the generated certificate/key pair during its validity period. This is to prevent multiple apps running behind a common endpoint address to use different certificates.

[mp911de]

Created showcase in mp911de/spring-cloud-vault-config, Branch issue/18-ssl-certificate-generation

[mp911de]

See also https://github.com/spring-projects/spring-boot/issues/6148

[czerwina]

Hi @mp911de is the showcase code good enough to use in a prod environment? As a security enhancement were wanting to move to ssl between the boot microservices internally below zuul and certificates issued by vault would be a good solution

[mp911de]

Issuing SSL certificates with Vault is a very neat feature but imposes certain challenges that IMHO should be addressed before we can provide SSL certificate generation with Spring Cloud Vault.

The showcase code works for SSL certificate issuing by providing a hostname externally. The code requests a certificate and stores it inside Vault to not generate a certificate upon each start as long as a previously issued certificate exists and is valid. The code does not solve the issue with concurrent application startup; Race conditions are possible if two instances start up at the same time which can lead to two certificates issued. That is a problem especially when you run two services behind a common DNS name – each service uses different private and public keys.

I wouldn't want to rely on the code because of the mentioned concurrency issues. A possible solution could be the integration of cluster coordination (leader election) but that again requires access to particular backends (Zookeeper, Hazelcast, Redis,…) which adds lots of complexity that could be probably solved in a different way.

[czerwina]

Vault can store keys etc in consul as a storage layer in a ha configuration. Would that mitigate some of the issues? In our case, as long as the certificate is valid within the infrastructure it doesn't matter if it's a different certificate (per microservice)

[mp911de]

I added a naïve example for SSL auto-configuration. There are some gaps (like HPKP and startup synchronization). Besides that, it auto-configures the embedded web server.

Example code: spring-cloud-vault-config-samples/pki.

[kautkata]

Shame that this issues hasn't had any progress for what 3 years.

We are setting up micro service based solution which of cause will run in containers (swarm possibly k8) but we don't entirely trust swarms native network traffic encryption (key rotation, actual cryptographic strength and stuff) but we need to run our solution in network with questionable security level. (customer on-prem deployments - you may never know how good/bad at infrastructure setup they are)

From this perspective it would be nice to delegate all TLS certificate generation to Vault as it will know better (than customers with questionable skill level) how to generate and securely distribute TLS certificates.

[mp911de]

Hi @kautkata,

Thanks for the ping. While certificate generation seems a straight-forward task (and from a single-node perspective, it's exactly that), the challenges arise from synchronization, expiration and certificate reuse.

Applications that request individually a certificate on startup do not reuse certificates that might have been valid for a longer time period. Apps exposed under the same service name would use different keys (which is discouraged as such an experience indicates a compromised system). Also, there is no easy way to switch a certificate while the server is running. The connector needs rebinding, more likely a restart/reconfiguration of the connector and so we're down to server implementation specifics.

That being said, a non-production ready implementation is trivial. We're happy to collaborate on a production-ready arrangement for web server SSL certificate generation so feel free to chime into this discussion or come up even with a pull request.

[kautkata]

Hello @mp911de ,

I'll just share my thoughts, please keep in mind that I'm wrong a lot.

Certificate (non)reuse issue, is this really an issue ? At the end of the day underlying purpose for communication over TLS channel is to make impossible to read sniffed network traffic. From attackers perspective you sniff traffic find (somewhere) private key, decrypt captured traffic and your happy. So from this perspective if certificate get's reused attackers job is made a lot easier as on private/public key pair is being reused. This in essence is similar to "wild card certificates are bad" concept - if you have for instance 50 web servers running across huge infrastructure and they all use one and the same wildcard cert then it's private/public key strength is only as god as security setup on the weakest server. If one server is compromised and certificate leaks then you need to change cert on all 50 of them, so from this perspective new certificate generation on each app startup is fine (taking into account Vaults pki role no_store parameter).

From clients perspective if HSTS and stuff is disabled it should be fine as client should trust CA not terminal certificate. On the other hand if client performs terminal certificate pinning then you shouldn't use dynamically generated certificates at all

[mp911de]

Certificate (non)reuse issue, is this really an issue ? At the end of the day underlying purpose for communication over TLS channel is to make impossible to read sniffed network traffic.

There are other assumptions for TLS such as authenticity, consistent keying across multiple nodes and indeed encryption.

This in essence is similar to "wild card certificates are bad" concept - if you have for instance 50 web

I agree that reusing the same certificate across multiple external hostnames isn't ideal. However, if you loadbalance across multiple nodes behind a single external hostname, that's a different issue.

[bendem]

For the sake of the argument, what is wrong with having multiple upstreams use multiple certificates as long as they are valid?

If you terminate the TLS connection at the loadbalancer, the client never sees the upstream cert and the load balancer has at least one connection (thus one handshake) per upstream, I don't see a problem here.

If you don't terminate the TLS connection, I can see the problem happening if you don't have sticky connections, if you have sticky connections, clients will be just fine so long as the upstream pool doesn't change too often.

Is the non TLS-terminating loadbalancer without sticky connection a large enough use case that it should prevent other use cases from being met? Also, considering this, the implementation will be much much simpler (start, request certificate, renew, stop[, revoke]).

Maybe I'm missing something obvious here.