Negotiating TLS when using `authority` for Envoy routing

lennyburdette commented 5 years ago

Problem description

I'm trying to connect a grpc-node client to an upstream service through Envoy. We would prefer to connect to Envoy via unix sockets, but that's not currently possible. Instead, we're connecting via TCP and our security team requires that we use SSL.

Envoy uses the http/2 authority metadata to route requests to the correct upstream cluster. The grpc client is rejecting the request with an Invalid host upstream.service.host set in :authority metadata. because the Envoy SSL certificate CNs are specific to the local machine.

In the following code snippet:

Envoy is running on port 8888.
this.host points to the local machine.
this.host is the CN in Envoy's SSL certificate.
upstream.service.host is the service cluster name for the upstream service as Envoy knows it. It is not in the Envoy SSL certificate.

creds = grpc.credentials.createSsl(/* same certs as Envoy */);
client = new ServiceClient('this.host:8888', creds, {
  'grpc.default_authority': 'upstream.service.host',
  'grpc.ssl_target_name_override': 'this.host'
});
client.getFoo(new GetFooRequest(), (err, resp) => { ... });

I hoped that grpc.ssl_target_name_override would override the authority check (based on this comment for the python client), but that does not appear to be the case. Is there another way to skip the authority check in the grpc client? Or is there another way to securely proxy grpc traffic through Envoy?

Environment

OS: CentOS 7
Node version: 10.13.0
Package name and version: gRPC@1.16.0

Additional context

I executed the above code with GRPC_VERBOSITY=debug GRPC_TRACE=all enabled. Here is the output with identifying information redacted: https://gist.github.com/lennyburdette/383321f4301e24e53e6cbe2f04381e89

Thank you!

nicolasnoble commented 5 years ago

grpc.ssl_target_name_override definitely is supposed to work.

Your code snippet however is strange to me. 'this.host' with quotes?

lennyburdette commented 5 years ago

Ack sorry, I can see how that would be confusing. I'm actually doing this:

client = new ServiceClient(`${require('os').hostname()}:8888', creds, {
  'grpc.default_authority': 'upstream.service.host',
  'grpc.ssl_target_name_override': require('os').hostname()
});

nicolasnoble commented 5 years ago

Right. I've read a bit more the whole issue and log actually. This is what I get when casually looking over github issues during the week-end.

The ssl_target_name_override can only be used to override the check done on the TCP connecting hostname, not on anything else. Basically, this is used if you're telling grpc to connect to host 'X', but make your SSL certificate checks on this hostname as if you actually asked to connect to host 'Y'. If anything, this is for testing purposes, where you're providing mock certificates when connecting somewhere. This isn't going to override anything else, and certainly not the authority. Things need to match, that is.

In your case, this would only be relevant if you were to do this for instance:

client = new ServiceClient('localhost:8888', creds, {
  'grpc.default_authority': 'upstream.service.host',
  'grpc.ssl_target_name_override': 'some.real.hostname.matching.localhost.certificate'
});

lennyburdette commented 5 years ago

That makes sense. Sounds like we're stuck, seeing as TLS and the authority metadata are inextricably tied together. This isn't a problem for golang and java clients as the grpc bindings support unencrypted traffic over a unix socket.

The only alternative I can come up with is configuring Envoy to use a different way of specifying the upstream cluster that's not related to security, like a Host header. If you know of any other solutions, I'm all ears. I'll close the ticket in the meantime.

Thanks for responding quickly on a weekend!

nicolasnoble commented 5 years ago

We haven't done any work on adding support for unix domain sockets because of the low demand for it and the work necessary to add said support. We can however consider re-prioritizing this feature if there's a good case for it. Envoy might be it, but I don't necessarily know all the context to fully make the case internally.

lennyburdette commented 5 years ago

We (Square) run our microservices on our own hardware and haven't moved to container orchestration like Kubernetes yet. We run the Envoy process as a "sidecar" alongside the microservice process and send HTTP traffic between them over unix sockets. Envoy terminates TLS, abstracting away transport security and service mesh concerns from the application developers.

As I understand it, unix sockets provide better security for the "sidecar" service model because you can lock them down to a particular user with file permissions. This is unlike TCP sockets, where any user can listen to traffic, making them less secure unless you add TLS.

Supporting unix sockets in libuv would totally unblock support for grpc in Node.js for us. I'm also open to trying out the pure JS library if you can point me to a protoc plugin for static codegen that uses @grpc/grpc-js. Thanks again!

grpc / grpc-node