Mutually authenticated TLS to backends

[sorenisanerd]

Thanks for submitting an issue to gorouter. We are always trying to improve! To help us, please fill out the following template.

Issue

Communication between gorouter and backends is unauthenticated and unencrypted.

Context

Security policies may require all connections to be mutually authenticated as well as encrypted. As gorouter<->backend communication is in clear text, Cloud Foundry as a whole may fail to meet security policies.

Steps to Reproduce

Deploy an application to cloud foundry. Make a request to it.

Expected result

Applications should be able to specify that they expect SSL connections in which case the gorouter should create a TLS connection and present a client certificate. The backend should be able to validate the client certificate against a CA cert (either to be found on the filesystem somehow or in an environment variable).

Current result

Connections from gorouter are in clear text.

Possible Fix

As part of an application's manifest, it should be able to specify that it expects SSL connections. This setting will be passed on to the gorouter which will then make SSL connections to the given application.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/145913531

The labels on this github issue will be updated when the story is started.

[sorenisanerd]

When or if this lands, we need to augment the API to let applications tell Cloud Foundry that they're expecting SSL. I don't have a very good overview of the pieces involved there, but the CLI, the cloud controller, and the route-emitter at least will need to be updated.

[shalako]

Hello @sorenh

In #167 you asked that the Gorouter forward the client cert, and/or some data in the cert, to the backend application. Do you need both of these or only one?

Have you considered the performance implications for having the Gorouter negotiate an mTLS handshake for every request? Based on previous testing, I would expect this to increase latency by 300%.

Have you considered using IPSec to secure traffic between components in the platform?

Have you considered using TCP Routing for applications with these requirements? As the TCP Router passes through a L4 connection, the request would be encrypted all the way to the app, and the app would receive the client cert.

A colleague at Pivotal is working on coordinating a call for us to discuss these requirements with you.

[sorenisanerd]

Both are needed. Without #167 , the backend has no clue who it's talking to.

I understand there will be performance implications. Pardon the snark, but handing highly confidential information over to eavesdroppers isn't more acceptable just because we can do it in a jiffy.

IPSec -- if done well -- could get us part of the way. We would at least know that we're talking to the correct host, but there's no cryptographic validation that we're talking to the correct service. Nevertheless, the IPSec add-on provided by Pivotal uses the same certificate and private key on each node, so we wouldn't even get host level authentication, only cluster level.

We're resorting to the TCP router for now, but it makes discoverability vastly more complicated (simple DNS lookup vs. either SRV DNS lookups (which isn't provided out of the box anyway, so we'd need a 3rd party component to populate a DNS server with route information) or something akin to Eureka), we can't use route services for logging, throttling, malware scanning, etc., we can't route to different apps based on path.... I could go on. The gorouter is a very important part of Cloud Foundry's value proposition, IMO.

[shalako]

Is it fair to say that #167 represents a need for apps to obtain some information from the originating client cert, and that this represents a need for encryption from client to app? This issue proposes a specific solution, to what might be a requirement that could be solved other ways.

If all you need is encryption between Gorouter and Diego Cell, then IPSec should suffice. Is it not sufficient that the router can only connect to backends known to be in a trusted pool? Why should identity of the backend matter to Gorouter?

[sorenisanerd]

Before I go any further with this, I should say that I've totally been using the wrong term here. I thought mtls was "mutual TLS". What I mean is mutually authenticated TLS, so TLS where the client also presents a certificate.

Darn, I have a lot of clarification to do.

[sorenisanerd]

Yes, #167 is to provide information about the client to the application.

This issue is not just about encryption. Encryption is necessary, but not sufficient. Especially since the IPSec addon shares the private key across all VM's, but even if it didn't, you only get host-level authentication of the endpoints. Our security policy requires mutual authentication of the endpoints, not just the hosts.

If you can think of other ways to achieve that, I'm all ears.

[shalako]

I should say that I've totally been using the wrong term here. I thought mtls was "mutual TLS". What I mean is mutually authenticated TLS, so TLS where the client also presents a certificate.

I don't think you've been using the wrong term. This is also my understanding of mTLS.

[shalako]

I expect your policy of mTLS is between the client and server. Gorouter is not the client. As a simple example, the client and server are both microservices A and B, one talking to the other.

A (client) ---> LB ---> Gorouter ---> B (service)

B's cert may be hosted by a load balancer or Gorouter. The tricky part of mTLS is providing the client cert for A to B for validation. As long as that occurs, you have mTLS between A and B.

To secure the leg between Gorouter and B, you could encrypt that traffic with IPSec. This shouldn't require mTLS between Gorouter and B; what is important is the client cert of A, not Gorouter.

I don't understand your requirement about host vs endpoint authentication.

[sorenisanerd]

Oh, I should mention that we need the gorouter's client cert presented to route services as well.

[sorenisanerd]

No, really, this is about mTLS between the gorouter and the backend. When the gorouter terminates my SSL connection in order to inspect the request and determine the target application and thus the target backend, the connection to the backend needs to be mutually authenticated.

IPSec does not give me that authentication. IPSec done correctly() ensures that I'm talking to the right host*. As the backend, I can't tell if I'm actually talking to the gorouter or some other client running on the same VM as the gorouter. As the gorouter, I can't tell if I'm talking to the correct backend or some other application that somehow managed to hijack the port.

(*): The IPSec addon provided by Pivotal doesn't even help me validate that I'm talking to the right host, because all VM's share the same keypair.

[shashwathi]

@sorenh : This is being currently address in epic for route integrity. Let us know if this solves the problem discussed here.

Regards Shash

[shubhaat]

Closing this, as Gorouter supports mTLS with backends. The Envoy run in containers does not yet request a client cert, but Gorouter can be configured to provide one using properties router.backends.cert_chain and router.backends.private_key.

@emalm, PM for Diego, is gathering evidence in support of configuring the Envoy to request/require a client cert from Gorouter. Issues can be added or tracked at https://github.com/cloudfoundry/diego-release/issues.